Develop Reference

Extracting specific JSON values in python

JSON return from spotify api. Example:
{
"tracks": {
"href": "https://api.spotify.com/v1/search?query=Stero+Hearts&type=track&offset=0&limit=1",
"items": [
{
"album": {
"album_type": "album",
"artists": [
{
"external_urls": {
"spotify": "https://open.spotify.com/artist/4IJczjB0fJ04gs4uvP0Fli"
},
"href": "https://api.spotify.com/v1/artists/4IJczjB0fJ04gs4uvP0Fli",
"id": "4IJczjB0fJ04gs4uvP0Fli",
"name": "Gym Class Heroes",
"type": "artist",
"uri": "spotify:artist:4IJczjB0fJ04gs4uvP0Fli"
}
]
}
}
]
}
}
Broken Code
import requests, json
spotifytrack = input("Name of Song?\\n")
link = "https://api.spotify.com/v1/search?q=" + spotifytrack + "&type=track&limit=1"
token = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
header = {
"Authorization": "Bearer {}".format(token),
"Content-Type": "application/json",
"Accept": "application/json",
}
auth_response = requests.get(link, headers=header)
pretty_response = json.dumps(auth_response.json(), indent=4)
data_by_user = {}
for d in auth_response:
data_by_user[d["artist"]] = d
print(data_by_user["uri"])
"""
def find_track_from_json(auth_response, artist):
return [p for p in auth_response if p["artist"] == artist][0]["uri"]
urii = find_track_from_json(auth_response, "uri")
print(urii)
x = load.json(auth_response.json())
print("Here is the data whic we have imported\n")
print(pretty_response)
print(x["name"])
print(x["uri"])
print(x["spotify"])
"""
Errors noticed:
File "spotify.py", line 19, in <module>
data_by_user[d["artist"]] = d
TypeError: byte indices must be integers or slices, not str
The aim is to convert word search to link in a cli application.
I tried load.json which i saw in some website and also tried def.
I expected the program to find out the artist name and uri from the json and print it in the cli interface.

You are iterating over the encoded json string:
auth_response = requests.get(link, headers=header)
for d in auth_response:
Python is complaining that you aren't providing a numerical index, which is correct as auth_response is just a string!
You should call json.loads to decode the string, and then you can iterate over it.
auth_response = requests.get(link, headers=header)
decoded_auth_response = json.loads(auth_response)
data_by_user = {}
for d in decoded_auth_response:
data_by_user[d["artist"]] = d
As you haven't provided the full json output from the API call I'm not sure what data is actually in decoded_auth_response, and you haven't described what your expected output would look like, so you may need to do some more work to find the correct data in each iteration.

The result from requests.get() is a requests.Response object. As far as I can see you want to iterate over the response body which is JSON. The requests.Response object has a .json() method which returns a dict from the response JSON.
Looking at the response you would probably want to iterate over resp_json['tracks']['items'] which is a list.
So to summarize your code should look something like this:
auth_response = requests.get(link, headers=header)
items = auth_response.json()['tracks']['items']
for d in items:
print(d)

How to merge two api in python by id in python?

I want to create a dictionary with using 2 api. Can you guide me?
I want to create a dictionary with using 2 api. Can you guide me?
url = 'https://test.com/api/v1/'
tags = []
result=[]
response = requests.get(url)
results = json.loads(response.text)
for data in results['results']:
second API
url = 'https://test.com/api/v1/'+data['tags_id']
response = requests.get(url)
results = json.loads(response.text)
for data in results['tags']:
tags.append(data['title'])
result of first api
results: [
{
"title": "subject1",
"tags_id": "86111ae6",
},
{
"title": "subject2",
"tags_id": "86ae6",
}]
expected result
results: [
{
"title": "subject1",
"tags: ['a','b'],
},
{
"title": "subject2",
"tags": ['c','d','f'],
}]
second API
"tags": [
{
"title": 'a',
},
{
"title": 'b',
},
]

Since each "title" from the first API has exactly one "tags_id", I think what you want is something like:
url = 'https://test.com/api/v1/'
response = requests.get(url)
results = json.loads(response.text)
output = list()
for d in results['results']:
response = requests.get(f'https://test.com/api/v1/{d["tags_id"]}/show_tags')
result = json.loads(response.text)
output.append({"title": d["title"],
"tags": [t["title"] for t in result["tags"]]})

Getting more than 100 contacts with Qualtrics API using the nextPage URL

I successfully setup the Qualtrics API for exporting contacts from working with the example script from their website. My problem is the API only exports 100 contacts at a time. It seems if I use the the url from the nextPage field in my initial json export to do another API call I can get another 100 but that isn't ideal. I sometimes have lists with over 10,000 people and cannot manually work this.
I am a Python noob and would like to know how its possible to use the nextPage URL function to receive more than 100 responses at a time.
The API Call looks like this
# List Contacts in Mailing List
import requests
# Setting user Parameters
apiToken = "YOUR API TOKEN"
dataCenter = "YOUR DATACENTER"
directoryId = "POOL_123456"
mailingListId = "CG_123456"
baseUrl = "https://{0}.qualtrics.com/API/v3/directories/{1}/mailinglists/{2}/contacts".format(dataCenter, directoryId, mailingListId)
headers = {
"x-api-token": apiToken,
}
response = requests.get(baseUrl, headers=headers)
print(response.text)
And I receive a similar results to this with only 100 responses:
{
"meta": {
"httpStatus": "200 - OK",
"requestId": "7de14d38-f5ed-49d0-9ff0-773e12b896b8"
},
"result": {
"elements": [
{
"contactId": "CID_123456",
"email": "js#example.com",
"extRef": "1234567",
"firstName": "James",
"language": "en",
"lastName": "Smith",
"phone": "8005552000",
"unsubscribed": false
},
{
"contactId": "CID_3456789",
"email": "person#example.com",
"extRef": "12345678",
"firstName": "John",
"language": "en",
"lastName": "Smith",
"phone": "8005551212",
"unsubscribed": true
}
],
"nextPage": null
}
}
Does anyone have an idea how I can loop the nextPage information to get an entire list of contacts, no matter how many sets of 100 are contained? I have some lists where there are tens, hundreds, and thousands of contacts and would like it to work for all.
Appreciate all input! Thanks!

Use a while loop and rename baseUrl to nextPage:
nextPage = "https://{0}.qualtrics.com/API/v3/directories/{1}/mailinglists/{2}/contacts".format(dataCenter, directoryId, mailingListId)
while nextPage is not None:
response = requests.get(nextPage, headers=headers)
json_data = json.loads(response.text)
#process json_data
nextPage = json_data['result']['nextPage']

handle pagination recursively

I'm using requests lib to fetch data from remote server and I'm saving the data in the model, but I need to handle pagination, currently I'm only loading one page from server.
I have a url for pagination like so
{
"status": "success",
"count": 32,
"total": 32,
"next": "https://pimber.ly/api/v2/products?sinceId=5c3ca8470985af0016229b5b",
"previous": "https://pimber.ly/api/v2/products?maxId=5c3ca8470985af0016229b04",
"sinceId": "5c3ca8470985af0016229b04",
"maxId": "5c3ca8470985af0016229b5b",
"data": [
{
"Primary ID": "API_DOCS_PROD1",
"Product Name": "Example Product 1",
"Product Reference": "Example Reference 1",
"Buyer": "Example Buyer 1",
"_id": "5c3ca8470985af0016229b04",
"primaryId": "API_DOCS_PROD1"
},
I've tried to use python generator to handle current situation but, that does not do anything
_plimber_data = response.json()
yield _plimber_data
_next = _plimber_data['next']
print(_next)
for page in _next:
_next_page = session.get(_plimber_data, params={'next': page}).json()
yield _next_page['next']
for _data in page:
Product.objects.create(
qr_id=_data['primaryId'],
ean_code=_data['EAN'],
description=_data['Description105'],
category=_data['Category'],
marketing_text=_data['Marketing Text'],
bullet=_data['Bullet 1'],
brand_image=_data['Brand Image'],
image=_data['Images']
)
logger.debug(f'Something went wrong {_data}')
print(f'This is the Data:{_data}')
Can someone please explain me how to handle this so I can load all the data into the database, thanks.

Ok so I've solved it, two thinks first generator function
def _get_product():
"""
TODO: Fetch data from server
"""
headers = {
'Accept': 'application/json',
'Content-Type': 'application/json',
'Authorization': settings.TOKEN
}
try:
response = requests.get(
url=f'{settings.API_DOMAIN}',
headers=headers
)
response.raise_for_status()
except HTTPError as http_err:
print(f'HTTP error occurred: {http_err}')
else:
_plimber_data = response.json()
while _plimber_data['next'] is not None:
response = requests.get(
_plimber_data['next'],
headers=headers
)
_plimber_data = response.json()
for _data in _plimber_data['data']:
yield _data
Then I iterate over generator function, and save the data:
def run(self):
_page_data = _get_product()
for _product in _page_data:
Product.objects.create(
qr_id=_product['primaryId'],
ean_code=_product['EAN'],
description=_product['Description105'],
category=_product['Category'],
marketing_text=_product['Marketing Text'],
bullet=_product['Bullet 1'],
brand_image='\n'.join(_product['Brand Image']),
image='\n'.join(_product['Images'])
)
logger.debug(f'Something went wrong {_product}')
print(f'This is the Data:{_product}')

JSONDecodeError: Expecting value: line 1 column 1 (char 0) error

I am facing this error while making request to fetch json from api.
I can get json data using the "/v1/articles' path.
conn = http.client.HTTPSConnection("api.xxxx.com.tr")
headers = {
'accept': "application/json",
'apikey': "cd6b6c96799847698d87dec9f9a731d6"
}
filter = "daily"
conn.request("GET", "/v1/articles", headers=headers)
reader = codecs.getreader("utf-8")
res = conn.getresponse()
data = json.load(reader(res))
json.dumps(data)
return data
But i am having JSONDecodeError if i set filter. Code:
conn = http.client.HTTPSConnection("api.xxxx.com.tr")
headers = {
'accept': "application/json",
'apikey': "cd6b6c96799847698d87dec9f9a731d6"
}
conn.request("GET", "/v1/articles?$filter=Path eq '/daily/'", headers=headers)
reader = codecs.getreader("utf-8")
res = conn.getresponse()
data = json.load(reader(res))
json.dumps(data)
return data
I tried same filter using Postman with no error and i can get Json data.
Returned Json data from Postman:
[
{
"Id": "40778196",
"ContentType": "Article",
"CreatedDate": "2018-03-20T08:28:05.385Z",
"Description": "İspanya'da 2016 yılında çalınan lüks otomobil, şasi numarası değiştirilerek Bulgaristan üzerinden getirildiği Türkiye'de bulundu.",
"Files": [
{
"FileUrl": "http://i.xxxx.com/i/xxxx/98/620x0/5ab0c6a9c9de3d18a866eb54.jpg",
"Metadata": {
"Title": "",
"Description": ""
}
}
],
"ModifiedDate": "2018-03-20T08:32:12.001Z",
"Path": "/gundem/",
"StartDate": "2018-03-20T08:32:12.001Z",
"Tags": [
"ispanya",
"Araç",
"Hırsız",
"Dolandırıcı"
],
"Title": "İspanya'da çalınan lüks araç Türkiye'de bulundu!",
"Url": "http://www.xxxx.com.tr/gundem/ispanyada-calinan-luks-arac-turkiyede-bulundu-40778196"
}
]
I can not figure out the problem. It would be great if anyone help me about this issue. Thank you.

I finally figured out the problem! Using the requests library have solved my problem now I can filter the api request.
data = requests.get('https://api.xxxxx.com.tr/v1/articles', headers =
headers, params={"$filter":"Path eq '/xxxxxx/'"}).json()
I am leaving this answer here for anyone else who can need this solution in the future.
Thanks for all your suggestions.

The problem is in the following line
data = json.load(reader(res))
when your response is not a json string, JSONDecodeError occurs. so, add an additional logic to see if the response is None or a json string. First thing, print the reader(res) and see what the return is