Develop Reference

How do i use the "next" url for post request?

So how does python call the "next" page in a post request?
I know what i neeed to do, but not sure how to implement, all the examples in youtube use a # and not a cursor, being a semi beginner i am a bit confused
This is my code so far:
def main_request(headers, url1, params):
response = requests.post(url1, headers=headers, json=params, verify=False)
jsonData = response.json()
has_next_key = False
nextKey = ""
if "next_key" in jsonData:
next = True
nextKey = jsonData["next"]
while has_next_key:
data = {"limit_count":500, "limit_size":10000,"curr_key":nextKey}
params = {"data":json.dumps(data, separators=(",", ":"))}
req = requests.post(url1, headers=headers, json=params, verify=False) ## this should do GET request for the third page and so on...
if "next_key" in req:
nextKey = req["next_key"]
print(nextKey) # this returns "3321" which is the value for "next_key" in second page
else:
has_next_key = False
# no next_key, stop the loop
This is the value is brings back at the end of each request
{
"data": [],
"metadata": {},
"links": [
{
"href": "https://us.api.insight.rapid7.com:443/vm/v4/integration/assets?page=0&size=2",
"rel": "first"
},
{
"href": "https://us.api.insight.rapid7.com:443/vm/v4/integration/assets?page=0&size=2",
"rel": "self"
},
{
"href": "https://us.api.insight.rapid7.com:443/vm/v4/integration/assets?page=1&size=2&cursor=1542252837:::_S:::12474375-34a7-40a3-9821-28db0b5cc90e-default-asset-10",
"rel": "next"
},
{
"href": "https://us.api.insight.rapid7.com:443/vm/v4/integration/assets?page=1097&size=2",
"rel": "last"
}
]
}
according to rapid7 support, i need to use the cursor value

Given your jsonData as input, you can use this code to get the next url and assign it to url1:
for item in jsonData.get("links", []):
if item["rel"] == "next":
url1 = item["href"]
break
This only finds the first url. If you need all urls, I'd recommend adding all urls to a list.
For examples:
links = [item["href"] for item in jsonData.get("links", []) if item["rel"] == "next"]

Extracting specific JSON values in python

JSON return from spotify api. Example:
{
"tracks": {
"href": "https://api.spotify.com/v1/search?query=Stero+Hearts&type=track&offset=0&limit=1",
"items": [
{
"album": {
"album_type": "album",
"artists": [
{
"external_urls": {
"spotify": "https://open.spotify.com/artist/4IJczjB0fJ04gs4uvP0Fli"
},
"href": "https://api.spotify.com/v1/artists/4IJczjB0fJ04gs4uvP0Fli",
"id": "4IJczjB0fJ04gs4uvP0Fli",
"name": "Gym Class Heroes",
"type": "artist",
"uri": "spotify:artist:4IJczjB0fJ04gs4uvP0Fli"
}
]
}
}
]
}
}
Broken Code
import requests, json
spotifytrack = input("Name of Song?\\n")
link = "https://api.spotify.com/v1/search?q=" + spotifytrack + "&type=track&limit=1"
token = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
header = {
"Authorization": "Bearer {}".format(token),
"Content-Type": "application/json",
"Accept": "application/json",
}
auth_response = requests.get(link, headers=header)
pretty_response = json.dumps(auth_response.json(), indent=4)
data_by_user = {}
for d in auth_response:
data_by_user[d["artist"]] = d
print(data_by_user["uri"])
"""
def find_track_from_json(auth_response, artist):
return [p for p in auth_response if p["artist"] == artist][0]["uri"]
urii = find_track_from_json(auth_response, "uri")
print(urii)
x = load.json(auth_response.json())
print("Here is the data whic we have imported\n")
print(pretty_response)
print(x["name"])
print(x["uri"])
print(x["spotify"])
"""
Errors noticed:
File "spotify.py", line 19, in <module>
data_by_user[d["artist"]] = d
TypeError: byte indices must be integers or slices, not str
The aim is to convert word search to link in a cli application.
I tried load.json which i saw in some website and also tried def.
I expected the program to find out the artist name and uri from the json and print it in the cli interface.

You are iterating over the encoded json string:
auth_response = requests.get(link, headers=header)
for d in auth_response:
Python is complaining that you aren't providing a numerical index, which is correct as auth_response is just a string!
You should call json.loads to decode the string, and then you can iterate over it.
auth_response = requests.get(link, headers=header)
decoded_auth_response = json.loads(auth_response)
data_by_user = {}
for d in decoded_auth_response:
data_by_user[d["artist"]] = d
As you haven't provided the full json output from the API call I'm not sure what data is actually in decoded_auth_response, and you haven't described what your expected output would look like, so you may need to do some more work to find the correct data in each iteration.

The result from requests.get() is a requests.Response object. As far as I can see you want to iterate over the response body which is JSON. The requests.Response object has a .json() method which returns a dict from the response JSON.
Looking at the response you would probably want to iterate over resp_json['tracks']['items'] which is a list.
So to summarize your code should look something like this:
auth_response = requests.get(link, headers=header)
items = auth_response.json()['tracks']['items']
for d in items:
print(d)

Struggling to create a dictionary to store results from IBM Watson Entity Analysis

Am struggling to capture the results from the IBM Watson entity analysis in a dictionary. I would like to extract the sentiment of each link through a function. I have a function created to extract a single url. But the dictionary am trying to store the results captures only the last url results. I am new to Python, and appreciate any help.
Here is my entity analysis code,
# function to process an URL
def processurl(url_to_analyze):
# end point
endpoint = f"{URL}/v1/analyze"
# credentials
username = "apikey"
password = API_KEY
# parameters
parameters = {
"version": "2020-08-01"
}
# headers
headers = {
"Content-Type":"application/json"
}
# watson options
watson_options = {
"url": url_to_analyze,
"features": {
"entities": {
"sentiment": True,
"emotion": True,
"limit":10
}
}
}
# return
response = requests.post(endpoint,
data=json.dumps(watson_options),
headers=headers,
params=parameters,
auth=(username,password)
)
return response.json()
here is the function I created to pass the result from above
# create a function to extract the entities from the result data
def getentitylist(data,threshold):
result = []
for entity in data["entities"]:
relevance = float(entity["relevance"])
if relevance > threshold:
result.append(entity["text"])
return result
After looping through the URL's, I can't seemed to store the result in a dictionary so that I can pass that to my function for entity results
# method II: loop through news api urls and perform entity analysis and store it in a dictionary
entitydict = {}
for url in url_to_analyze:
entitydict.update(processurl(url))

I can't see where you are calling getentitylist, but in your url loop
entitydict = {}
for url in url_to_analyze:
entitydict.update(processurl(url))
update will be updating the dictionary based on key values. ie. this will overwrite the values for any keys already in the dictionary. As your response will look something like:
{
"usage": {
"text_units": 1,
"text_characters": 2708,
"features": 1
},
"retrieved_url": "http://www.cnn.com/",
"language": "en",
"entities": [
{
"type": "Company",
"text": "CNN",
"sentiment": {
"score": 0.0,
"label": "neutral"
},
"relevance": 0.784947,
"disambiguation": {
"subtype": [
"Broadcast",
"AwardWinner",
"RadioNetwork",
"TVNetwork"
],
"name": "CNN",
"dbpedia_resource": "http://dbpedia.org/resource/CNN"
},
"count": 9
}
]
}
The keys that will be updated are at the top level ie. usage, retrieved_url, retrieved_url, entities. So entitydict will only contain the response for the last url, as previous values for these keys will get overwritten.
What you should be doing is use the url as key to each response.
entitydict = {}
for url in url_to_analyze:
entitydict.update({url : processurl(url)})

copying data from json response [Python]

I have a scenario where I am trying to extract data from json response which is obtained from the GET request and then rebuilding the json data by changing some values and then sending a PUT request at same time after rebuilding the json data(i.e, after changing idter value)
below is the target json response.
target_json = {
"name": "toggapp",
"ts": [
1234,
3456
],
"gs": [
{
"id": 4491,
"con": "mno"
},
{
"id": 4494,
"con": "hkl"
}
],
"idter": 500,
"datapart": false
}
from the above json I am trying to change the idter value to my custom value and rebuild it into json data again and post the new json data.
Here is what I have tried :
headers = {'Authorization': 'bearer ' + auth_token, 'Content-Type':'application/json', 'Accept':'application/json'}
tesstid =[7865, 7536, 7789]
requiredbdy = []
for key in testid:
get_metadata_targetjson= requests.get('https://myapp.com/%s' %key, headers = headers)
metadata=get_metadata_target.json()
for key1 in metadata:
requiredbdy.append(
{
"metadata" : [{
"name": key1['name'],
"ts": key1['ts'],
"gs": key1[gs],
"idter": 100, #custom value which I want to change
"datapart": false
} ]
}
)
send_metadata_newjson= requests.put('https://myapp.com/%s' %key, headers = headers data = requiredbdy)
print(send_metadata_newjson.status_code)
Is this approach fine or How do I proceed in order to achieve this scenario.

You can use the built-in json module for this like so
import json
my_json = """
{
"name": "toggapp",
"ts": [
1234,
3456
],
"gs": [
{
"id": 4491,
"con": "mno"
},
{
"id": 4494,
"con": "hkl"
}
],
"idter": 500,
"datapart": false
}
"""
json_obj = json.loads(my_json)
json_obj['idter'] = 600
print(json.dumps(json_obj))
Prints
{"name": "toggapp", "ts": [1234, 3456], "gs": [{"id": 4491, "con": "mno"}, {"id": 4494, "con": "hkl"}], "idter": 600, "datapart": false}

There's this small script used it to find entries in some very long and unnerving JSONs. not very beautifull und badly documented but maybe helps in your scenario.
from RecursiveSearch import Retriever
def alter_data(json_data, key, original, newval):
'''
Alter *all* values of said keys
'''
retr = Retriever(json_data)
for item_no, item in enumerate(retr.__track__(key)): # i.e. all 'value'
# Pick parent objects with a last element False in the __track__() result,
# indicating that `key` is either a dict key or a set element
if not item[-1]:
parent = retr.get_parent(key, item_no)
try:
if parent[key] == original:
parent[key] = newval
except TypeError:
# It's a set, this is not the key you're looking for
pass
if __name__ == '__main__':
alter_data(notification, key='value',
original = '********** THIS SHOULD BE UPDATED **********',
newval = '*UPDATED*')

Retrieving data from JSON in python, and if object name matches, then store the key of that object

I am making REST calls on a server. The first REST call gets all the projects and from that I store the project's IDs in an array.
Below is the JSON.
For e.g. it would return something like this:
[
{
"expand": "description,lead,url,projectKeys",
"self": "http://localhost:8080/rest/api/2/project/10101",
"id": "10101",
"key": "GR1",
"name": "Group1Project",
"avatarUrls": {
"48x48": "http://localhost:8080/secure/projectavatar?avatarId=10324",
"24x24": "http://localhost:8080/secure/projectavatar?size=small&avatarId=10324",
"16x16": "http://localhost:8080/secure/projectavatar?size=xsmall&avatarId=10324",
"32x32": "http://localhost:8080/secure/projectavatar?size=medium&avatarId=10324"
},
"projectTypeKey": "software"
}
]
Then I'm looping through that array and making another REST call for each project id(10101).
This gives me groups/users against that project.
For example:
{
"self": "http://localhost:8080/rest/api/2/project/10000/role/10100",
"name": "Developers",
"id": 10100,
"actors": [
{
"id": 10207,
"displayName": "group2",
"type": "atlassian-group-role-actor",
"name": "group2",
"avatarUrl": "http://localhost:8080/secure/useravatar?size=xsmall&avatarId=10123"
}
]
}
I want to get all the project IDs where name == group2.
Following is my Python code for all of this but it's not working.
import requests
ids = []
response = requests.get('http://localhost:8080/rest/api/2/project',
auth=('*', '*'))
data = response.json()
for line in data:
ids.append(line["id"])
print(ids)
# Check if group exists in Project roles.
# If it does, then save the project name in the list of arrays.
projectNames = []
for id in ids:
url = 'http://localhost:8080/rest/api/2/project/'+id+'/role/10100'
response = requests.get(url,
auth = ('*', '*'))
data = response.json()
if data.displayName == 'group2':
projectNames.append(["id"])
Could you please help me out how to do this?
Thank you.

Tayyab,
You need to do this. It will work.
for actor in data['actors']:
if actor['displayName']=='group2':
projectNames.append(id)

projectNames = []
for id in ids:
url = 'http://localhost:8080/rest/api/2/project/'+id+'/role/10100'
response = requests.get(url,
auth = ('*', '*'))
data = response.json()
for actor in data["actors"]:
if actor["displayName"] and actor["displayName"] == "group2":
projectNames.append(actor["id"])

projectNames = set()
for id in ids:
url = 'http://localhost:8080/rest/api/2/project/'+id+'/role/10100'
response = requests.get(url,
auth = ('*', '*'))
data = response.json()
group2_actors = [actor['id'] for actor in data['actors']
if actor['displayName'] == 'group2']
if len(group2_actors) > 0:
projectNames.update(group2_actors)
projectNames is a set of unique actor ids with displayName == group2.

some_json = {}
result = [actor['id'] for actor in some_json['actors'] if actor['name']=='group2']
so result for that second json will be [10207]