Develop Reference

Loop values from response into url

I'm trying to create a loop on some response data from a website. The loop should effectively grab the value within the tag cursor, and read this into the url link and repeat this continuously until the last link has no tag cursor. The cursor tag has a value which gives info on the next url page that is loading.
I so far only get it to work individually likeso:
import requests
import pandas as pd
from io import StringIO
def ids(url):
payload={}
headers = {}
response = requests.request("GET", url, headers=headers, data=payload)
test2 = []
test= pd.read_json(StringIO(response.text), lines=True)
for m in test['meta'].items():
for j in m[1].values():
test2.append([j])
id = test2[1][0]
#split - the next part should repeat
html = f'https://webapi.depop.com/api/v2/search/products/?brands=1596&cursor={id}&itemsPerPage=24&country=gb&currency=GBP&sort=relevance'
r = requests.request("GET",html, headers=headers, data=payload)
list_id = []
read_id = pd.read_json(StringIO(r.text), lines=True)
for m in read_id['meta'].items():
for id in m[1].values():
list_id.append([id])
id2 = list_id[1][0]
return id2
Then I try:
url = 'https://webapi.depop.com/api/v2/search/products/?brands=1645&itemsPerPage=24&country=gb&currency=GBP&sort=relevance'
ids(url)
However, I have to replicate the code below 'split' in such a way that it can just loop the new id (so id2) into the f-string for the html and keep extracting the value within cursor and store the values as a big list. My approach I can only grab them one-by-one, and will have to keep repeating the code above which will take an enormous amount of time and code. Is there a more efficient way of doing this?
Expected output:
['MnwyNHwxNjQwMDA4MjI2', 'M3w0OHwxNjQwMDA4MjI2', 'NnwxMjB8MTY0MDAwODIyNg', ...]
Ways it may work:
Run the code within a while loop and have the f string take the last string in list id2, so that when the code iterates and produces a new id it then takes that latest id and so forth until the last 2 are equal we stop.
Aftering testing with a while loop it seems to work but I end up getting an error anyways:
def ids(url):
payload={}
headers = {}
response = requests.request("GET", url, headers=headers, data=payload)
list_id=[]
test= pd.read_json(StringIO(response.text), lines=True)
for m in test['meta'].items():
list_id.append(m[1]['cursor'])
while True:
html = f'https://webapi.depop.com/api/v2/search/products/?brands=1596&cursor={list_id[-1]}&itemsPerPage=24&country=gb&currency=GBP&sort=relevance'
r = requests.request("GET",html, headers=headers, data=payload)
read_id = pd.read_json(StringIO(r.text), lines=True)
for m in read_id['meta'].items():
try:
list_id.append(m[1]['cursor'])
except:
continue
if list_id[-2] == list_id[-1]:
break
return list_id
I get the following error right before the exception:
KeyError: 'meta'
But it still grabs the data when I check the list, how can I make it so that no error appears?

Python API gets invalid json

I have just started using python a few days ago and get work out the JSON format.
I use requests to get JSON data by API. However, I get the wrong decoded JSON format (JSON validator finds errors).
webpage = 'https://parser-api.com/parser/arbitr_api/run.php'
API = 'cant post it' #
output_results = []
cases = ['А65-22925/2017']
for i in cases:
params = {'key':API, 'CaseNumber':i}
results = requests.get(webpage, params = params)
output_results.append(results.text)
print (output_results)
with open ('file_name_case.json', 'w', encoding='utf8') as wr:
wr.write (str(output_results))
that is the snippet of the response that I get, which is wrong:
['{"Cases":[{"CaseId":"998ecaef-3da8-45ab-9f56-90bfc3375e11","CaseNumber":"\\u041065-22925\\/2017","CaseType":"\\u0410","Thirds":[],"Plaintiffs":[{"Name":"\\u041e\\u041e\\u041e \\"\\u0412\\u0420-\\u041f\\u043b\\u0430\\u0441\\u0442\\", \\u0433.\\u041a\\u0430\\u0437\\u0430\\u043d\\u044c","Address":"421001, \\u0420\\u043e\\u0441\\u0441\\u0438\\u044f, \\u0433.\\u041a\\u0430\\u0437\\u0430\\u043d\\u044c, \\u0420\\u0422, \\u0443\\u043b.\\u0421.\\u0425\\u0430\\u043a\\u0438\\u043c\\u0430, \\u0434.60, \\u043e\\u0444\\u0438\\u0441 164","Id":"dc26df83-6361-4de0-bc93-8c20ae0a4417"}],"Respondents":[{"Name":"\\u0424\\u0435\\u0434\\u0435\\u0440\\u0430\\u043b\\u044c\\u043d\\u0430\\u044f \\u0422\\u0430\\u043c\\u043e\\u0436\\u0435\\u043d\\u043d\\u0430\\u044f \\u0441\\u043b\\u0443\\u0436\\u0431\\u0430 \\u041f\\u0440\\u0438\\u0432\\u043e\\u043b\\u0436\\u0441\\u043a\\u043e\\u0435 \\u0442\\u0430\\u043c\\u043e\\u0436\\u0435\\u043d\\u043d\\u043e\\u0435 \\u0443\\u043f\\u0440\\u0430\\u0432\\u043b\\u0435\\u043d\\u0438\\u0435 \\u0422\\u0430\\u0442\\u0430\\u0440\\u0441\\u0442\\u0430\\u043d\\u0441\\u043a\\u0430\\u044f \\u0442\\u0430\\u043c\\u043e\\u0436\\u043d\\u044f, \\u0433.\\u041a\\u0430\\u0437\\u0430\\u043d\\u044c","Address":"420094, \\u0420\\u043e\\u0441\\u0441\\u0438\\u044f, \\u0433.\\u041a\\u0430\\u0437\\u0430\\u043d\\u044c, \\u0420\\u0422, \\u0443\\u043b.\\u041a\\u043e\\u0440\\u043e\\u043b\\u0435\\u043d\\u043a\\u043e, \\u0434.56","Id":"4b21e3e9-9d0c-42ce-bbec-4e1615e34698"}]...
the right format suppose to be like this:
{"Cases":[{"CaseId":"998ecaef-3da8-45ab-9f56-90bfc3375e11","CaseNumber":"\u041065-22925\/2017","CaseType":"\u0410","Thirds":[],"Plaintiffs":[{"Name":"\u041e\u041e\u041e \"\u0412\u0420-\u041f\u043b\u0430\u0441\u0442\", \u0433.\u041a\u0430\u0437\u0430\u043d\u044c","Address":"421001, \u0420\u043e\u0441\u0441\u0438\u044f, \u0433.\u041a\u0430\u0437\u0430\u043d\u044c, \u0420\u0422, \u0443\u043b.\u0421.\u0425\u0430\u043a\u0438\u043c\u0430, \u0434.60, \u043e\u0444\u0438\u0441 164","Id":"dc26df83-6361-4de0-bc93-8c20ae0a4417"}],"Respondents":[{"Name":"\u0424\u0435\u0434\u0435\u0440\u0430\u043b\u044c\u043d\u0430\u044f \u0422\u0430\u043c\u043e\u0436\u0435\u043d\u043d\u0430\u044f \u0441\u043b\u0443\u0436\u0431\u0430 \u041f\u0440\u0438\u0432\u043e\u043b\u0436\u0441\u043a\u043e\u0435 \u0442\u0430\u043c\u043e\u0436\u0435\u043d\u043d\u043e\u0435 \u0443\u043f\u0440\u0430\u0432\u043b\u0435\u043d\u0438\u0435 \u0422\u0430\u0442\u0430\u0440\u0441\u0442\u0430\u043d\u0441\u043a\u0430\u044f \u0442\u0430\u043c\u043e\u0436\u043d\u044f, \u0433.\u041a\u0430\u0437\u0430\u043d\u044c","Address":"420094, \u0420\u043e\u0441\u0441\u0438\u044f, \u0433.\u041a\u0430\u0437\u0430\u043d\u044c, \u0420\u0422, \u0443\u043b.\u041a\u043e\u0440\u043e\u043b\u0435\u043d\u043a\u043e, \u0434.56","Id":"4b21e3e9-9d0c-42ce-bbec-4e1615e34698"}]
Please help

You can do something like this:
import json
with open('file_name_case.json', 'w', encoding='utf-8') as f:
json.dump(data, f, ensure_ascii=False, indent=4)

Although Python syntax is very similar in many ways to JSON syntax it is not valid JSON to just str(<some python object>). You need to use the json module to write JSON.
Instead of taking results.text directly, use results.json() to decode the JSON response from the server.
Now you have a Python list of dicts as opposed to a Python list of strings containing JSON.
Then, as in kup's answer you can convert back to JSON:
output_results = []
for idx in cases:
params = {'key': API, 'CaseNumber': idx}
results = requests.get(webpage, params=params)
output_results.append(results.json())
with open('file_name_case.json', 'w') as fobj:
json.dump(output_results, fobj)

How can I access these json object in python

I'm making some data visualization from movies database api and I already access the data in the normal way but when i load the json data and for loop to print it, the data that out is just the column but I need to access the object inside.
url = "https://api.themoviedb.org/3/discover/movie?api_key="+ api_key
+"&language=en- US&sort_by=popularity.desc&include_adult=
false&include_video=false&page=1" # api url
response = urllib.request.urlopen(url)
raw_json = response.read().decode("utf-8")
data = json.loads(raw_json)
for j in data:
print(j)
i expect the output would be
[{'popularity': 15,
'id': 611,
'video': False,
'vote_count': 1403,
'vote_average': 8.9,
'title': 'lalalalo'},{....}]
but the actual output is
page
total_results
total_pages
results

The results are one level down. You are looping through the metadata.
Try changing your code to
import json
import urllib.request
api_key = "your api code"
url = "https://api.themoviedb.org/3/discover/movie?api_key=" + api_key +"&language=en- US&sort_by=popularity.desc&include_adult=false&include_video=false&page=1" # api url
response = urllib.request.urlopen(url)
raw_json = response.read().decode("utf-8")
data = json.loads(raw_json)
for j in data['results']:
print(j)
You need to change
data
to
data['results']

you can simply use requests module...
import requests
import json
your_link = " "
r = requests.get(your_link)
data = json.loads(r.content)
You shall have the json loaded up, then use your key "results" ["results"] and loop through the data you got.

Python JSONDecodeError: Expecting value: line 1 column 1

I got an error : JSONDecodeError: Expecting value: line 1 column 1 (char 0). But don't understand why.
Here is my code :
import json
import urllib.request
url = "apiurl"
data = json.loads(url)
# Open the URL as Browser, not as python urllib
page = urllib.request.Request(url,headers={'User-Agent': 'Mozilla/5.0'})
infile = urllib.request.urlopen(page).read()
data = infile.decode('ISO-8859-1') # Read the content as string decoded with ISO-8859-1
command_obj = {x['command']: x for x in data}
with open('new_command.json', 'w') as f:
json.dump(command_obj, f, indent=2)
With this fonction, i'm just trying to fetch data from an api and modify its format. Thanks for your help

You're trying to read the URL itself (and not its content) as JSON:
data = json.loads(url)
... instead you want to read the content returned from the API as JSON:
# Open the URL as Browser, not as python urllib
page = urllib.request.Request(url,headers={'User-Agent': 'Mozilla/5.0'})
infile = urllib.request.urlopen(page).read()
data = infile.decode('ISO-8859-1')
# avoid re-using `data` variable name
json_data = json.loads(data)
However, be aware that JSON should always be returned as UTF-8, never as ISO-8859-1 / latin-1.

How to get a specific value in a json string in python 3.6

I am trying to get a certain value in a string of json but I can't figure out how exactly to do it. I don't want to convert it into a string and strip / replace the unwanted pieces because then I won't be able to get the other values. My current code is:
username = "Dextication"
url = f"https://minecraft-statistic.net/api/player/info/{username}/"
response = requests.get(url)
json_data = json.loads(response.text)
print(json_data)
Edit:
when I run this, json.data = "{"status":"ok","data":{"online":0,"total_time_play":46990,"last_play":1513960562,"license":1,"name":"Dextication","uuid":"74d57a754855410c90b3d51bc99b8beb"}}"
I would like to only print the value: 46990

Try below code
import json, requests
username = "Dextication"
url = f"https://minecraft-statistic.net/api/player/info/{username}/"
response = requests.get(url)
json_data = json.loads(response.text)
result = json_data['data']['total_time_play']
print (result)