Develop Reference

Python web scraping nested dict key pairs - AttributeError

I'm attempting to scrape PGA stats from the API below.
url = 'https://statdata.pgatour.com/r/stats/current/02671.json?userTrackingId=exp=1594257225~acl=*~hmac=464d3dfcda2b2ccb384b77ac7241436f25b7284fb2eb0383184f48cbdff33cc4'
response = requests.get(url)
pga_stats = response.json()
I would like to only select the nested keys identified in this image. I've been able to traverse to the 'year' key with the below code, but I receive the following AttributeError for anything beyond that.
test = pga_stats.get('tours')[0].get('years')
(prints reduced dictionary)
test = pga_stats.get('tours')[0].get('years').get('stats')
'list' object has no attribute 'get'
My end goal is to write this player data to a csv file. Any suggestions would be greatly appreciated.

pga_stats.get('tours')[0].get('years') returns a list, not a dict. You actually want to use the get method on it's first element, like this:
test = pga_stats.get('tours')[0].get('years')[0].get('stats')

Python http request and loop over contents of JSON

I'm trying to learn Python and have following problem:
I get an error while running this as it cannot see the 'name' attribute in data.
It works when I grab one by one items from JSON. However when I want to do it in a loop it fails.
I assume my error is wrong request. That it cannot read JSON correctly and see attributes.
import requests
import json
def main():
req = requests.get('http://pokeapi.co/api/v2/pokemon/')
print("HTTP Status Code: " + str(req.status_code))
print(req.headers)
json_obj = json.loads(req.content)
for i in json_obj['name']:
print(i)
if __name__ == '__main__':
main()

You want to access the name attribute of the results attribute in your json_object like this:
for pokemon in json_obj['results']:
print (pokemon['name'])
I was able to guess that you want to access the results keys because I have looked at the result of
json_obj.keys()
that is
dict_keys(['count', 'previous', 'results', 'next'])

Because all pokemons are saved in a list which is under keyword results, so you firstly need to get that list and then iterate over it.
for result in json_obj['results']:
print(result['name'])

A couple things: as soon mentioned, iterating through json_obj['name'] doesn't really make sense - use json_obj['results'] instead.
Also, you can use req.json() which is a method that comes with the requests library by default. That will turn the response into a dictionary which you can then iterate through as usual (.iteritems() or .items(), depending if you're using Python 2 or 3).

Manipulate JSON object with non ASC-II characters?

I'm trying to make an API call to Google CSE from python and then manipulate the resulting object into a dictionary object that I can manipulate. I think this question is not duplicated because the issue here I believe is that there are non ASC-II characters which leads to the resulting object being of type 'NoneType' and the resulting json object 'null'. I've played with the options documented for json including "ensure_ascii=False", but haven't been successful. Any help will be greatly appreciated!
Code:
import pprint, os, json
from googleapisclient.discovery import build
def search(searchkey,datekey,developkey,enginekey):
service = build("customsearch", "v1",
developerKey=developkey).cse().list(
q=searchkey,dateRestrict=datekey,
cx=enginekey,
).execute()
pprint.pprint(service)
mykey = 'My_Private_Key'
myengine = '009333857041890623793:z_drq9obxp0'
object2write = search('narco','20170101-20170201',mykey,myengine)
type(object2write)
jsonAbder = json.dumps(object2write, ensure_ascii=False, allow_nan=False)
print(jsonAbder)

The proximal cause of your error is that your search function doesn't have an explicit return statement. Thus, it implicitely returns None, which gets encoded into JSON null. Your issue has nothing to do with character encodings.
Just add:
return service
at the end of your function.

Get value for only one JSON variable

I want to get the value for last with Python, using requests and json. I have this:
import requests
import json
res = requests.get('https://www.bitstamp.net/api/v2/ticker/btcusd/')
print res.json['last']
But it showed me this:
TypeError: 'instancemethod' object has no attribute '__getitem__'
It's working to get all JSON code with res.json() but I want only the value from 'last'.

res.json() will parse the responses json object, which you need to do, to access any value of it. So to access the value last, you need to do: res.json()['last']

trouble scraping from JSONP feed

I asked a similar question earlier
python JSON feed returns string not object
but I am having a little more trouble and don't understand it.
For about half of the dates this works and returns a JSON object
for example November 9 2013 works
url = 'http://data.ncaa.com/jsonp/scoreboard/basketball-men/d1/2013/11/09/scoreboard.html?callback=c'
r = requests.get(url)
jsonObj = json.loads(r.content[2:-2])
but if I try November 11 2013:
url = 'http://data.ncaa.com/jsonp/scoreboard/basketball-men/d1/2013/11/11/scoreboard.html?callback=c'
r = requests.get(url)
jsonObj = json.loads(r.content[2:-2])
I get this error
ValueError: No JSON object could be decoded
I dont understand why. When I put both urls into a browser they look exactly the same.

The JSON in the second feed is, in fact, invalid JSON. Found this by removing the callback function and running it through: http://jsonlint.com/
To see for yourself, search for the following ID: 336252
The lines just above that ID contain two commas in a row, which is disallowed by the JSON spec.
My guess is that the server at data.ncaa.com is trying to generate JSON itself rather than using a JSON library. You should contact the site administrator and make them aware of this error.

Using demjson
demjson.decode(r.content[2:-2])
seems to work