This question already has answers here:
HTTP requests and JSON parsing in Python [duplicate]
(8 answers)
Closed 7 years ago.
I am creating a Django web app.
There is a function which creates a JSON response like this:
def rest_get(request, token):
details = Links.get_url(Links, token)
result={}
if len(details)>0:
result['status'] = 200
result['status_message'] = "OK"
result['url'] = details[0].url
else:
result['status'] = 404
result['status_message'] = "Not Found"
result['url'] = None
return JsonResponse(result)
And I get the response in the web browser like this:
{"status": 200, "url": "http://www.bing.com", "status_message": "OK"}
Now from another function I want to consume that response and extract the data out of it. How do I do it?
You can use the json library in python to do your job. for example :
json_string = '{"first_name": "tom", "last_name":"harry"}'
import json
parsed_json = json.loads(json_string)
print(parsed_json['first_name'])
"tom"
Since you have created a web app. I am assuming you have exposed a URL from which you can get you JSON response, for example http://jsonplaceholder.typicode.com/posts/1.
import urllib2
import json
data = urllib2.urlopen("http://jsonplaceholder.typicode.com/posts/1").read()
parsed_json = json.loads(data)
The urlopen function sends a HTTP GET request at the given URL. parsed_json is a variable of the type map and you can extract the required data from it.
print parsed_json['userId']
1
The answer I want to suggest is a little different. In your scenario - where one function needs to be accessed from both server and client end, I would suggest provide some extra parameter and change the output based on that. This reduces overheads and unnecessary conversions.
For example, if you pass in an extra parameter and change the result like this, you don't need JSON parsing on python. Of course there are solutions to do that, but why need converting to json and then parsing back when you can avoid that totally?
def rest_get(request, token, return_json=True):
details = Links.get_url(Links, token)
result={}
if len(details)>0:
result['status'] = 200
result['status_message'] = "OK"
result['url'] = details[0].url
else:
result['status'] = 404
result['status_message'] = "Not Found"
result['url'] = None
if return_json: # this is web response, so by default return_json = True
return JsonResponse(result)
return result
Then in your python code call like this -
rest_get(request, token, return_json=False): # we are passing False, so the return type is dictionary and we can use it right away.
Related
This question already has answers here:
What is JSONP, and why was it created?
(10 answers)
Django - Parse JSONP (Json with Padding)
(2 answers)
Closed last month.
I'm using the API of an affiliate network (Sovrn), expecting to retrieve a product's specification using the URL.
As per their documentation, I use:
url = 'URL-goes-here'
headers = {
"accept": "application/json",
"authorization": "VERY-HARD-TO-GUESS"
}
response = requests.get(url, headers=headers)
The code is working, the response I get is 200, the header contains the magical content-type application/json line
when I do
print(response.text)
I get
NULL({"merchantName":"Overstock","canonicalUrl":"URL goes here","title":"product name",...});
I tested for response type of response.text, it's <class 'str'> as expected. But when I try to process the response as json:
product_details = json.load(response.text)
I get an error message:
requests.exceptions.JSONDecodeError: [Errno Expecting value]
I'm new to JSON, but I assume the error is due to the outer NULL that the (seemingly valid) data is wrapped in.
After spending a few hours searching for a solution, it seems that I must be missing something obvious, but not sure what.
Any pointers would be extremely helpful.
That's clearly a bug in the API. Assuming it will be fixed after you complain, you could add a hack to your code
def sovrn_json_load_hack(json_text):
"""sovrn is returning invalid json as of (revision here)."""
if not json_text.startswith ("NULL("):
return json.loads(json_text)
else:
return json.loads(json_text[5:-2])
You can ignore NULL( at the beginning and ); at the end by using string slicing:
product_details = json.loads(response.text[5:-2])
Additionally, you should be using json.loads() as the content is a string.
This question already has answers here:
How can I access the nested data in this complex JSON, which includes another JSON document as one of the strings?
(3 answers)
Closed 5 months ago.
This is my code:
req = urllib.request.Request(url, headers=hdr)
req.get_method = lambda: 'GET'
response = urllib.request.urlopen(req)
### Response: 200 OK
print("If Response=200 Script run is OK : " , response.getcode())
print(response.read())
and the result is:
If Response=200 Script run is OK: 200
b'[{"id":"d85b3704-60c9-XXXXXXX-2886629c732e","name":"XXXXXX-d85b37","location":"Trial","accountType":"Trial","url":"/www.ai/","accessToken":"eyJhbGciOiJodHRwOi8vd3d3LnczLm9yZy8yMDAxLzA0L3htbGRzaWctbW9yZSNobWFjLXNoYTI1NiIsInR5cCI6IkpXVCJ9.eyJBY2NvdW50SWQiOiJkODViMzcwNC02MGM5LTQ3NGYtYWVjYy0yODg2NjI5YzczMmUiLCJQZXJtaXNzaW9uIjoiQ29udHJpYnV0b3IiLCJFeHRlcm5hbFVzZXJJZCI6IjExNjAyMDI3NDYxNTkyMzc5NTU5MCIsIlVzZXJUeXBlIjoiR29vZ2xlIiwiSXNzdWVyTG9jYXRpb24iOiJUcmlhbCIsIm5iZiI6MTY2MzUzMzA4OSwiZXhwIjoxNjYzNTM2OTg5LCJpc3MiOiJodHRwczovL2FwaS52aWRlb2luZGV4ZXIuYWkvIiwiYXVkIjoiaHR0cHM6Ly9hcGkudmlkZW9pbmRleGVyLmFpLyJ9.VDoLlEdExAB0xMBP_pB6oN2DBL-9BQS0vEhpAq8d2o0","moveToArmStartedDate":"0001-01-01T00:00:00"}]'
How can I just filter and show only 'accessToken' in the output?
Pretty Simple:
resp: bytes = b'[{"id":"d85b3704-60c9-XXXXXXX-2886629c732e","name":"XXXXXX-d85b37","location":"Trial","accountType":"Trial","url":"/www.ai/","accessToken":"eyJhbGciOiJodHRwOi8vd3d3LnczLm9yZy8yMDAxLzA0L3htbGRzaWctbW9yZSNobWFjLXNoYTI1NiIsInR5cCI6IkpXVCJ9.eyJBY2NvdW50SWQiOiJkODViMzcwNC02MGM5LTQ3NGYtYWVjYy0yODg2NjI5YzczMmUiLCJQZXJtaXNzaW9uIjoiQ29udHJpYnV0b3IiLCJFeHRlcm5hbFVzZXJJZCI6IjExNjAyMDI3NDYxNTkyMzc5NTU5MCIsIlVzZXJUeXBlIjoiR29vZ2xlIiwiSXNzdWVyTG9jYXRpb24iOiJUcmlhbCIsIm5iZiI6MTY2MzUzMzA4OSwiZXhwIjoxNjYzNTM2OTg5LCJpc3MiOiJodHRwczovL2FwaS52aWRlb2luZGV4ZXIuYWkvIiwiYXVkIjoiaHR0cHM6Ly9hcGkudmlkZW9pbmRleGVyLmFpLyJ9.VDoLlEdExAB0xMBP_pB6oN2DBL-9BQS0vEhpAq8d2o0","moveToArmStartedDate":"0001-01-01T00:00:00"}]'
import json
data = json.loads(resp)
print(data[0]['accessToken'])
Okay, how does this work now? We load the response first into a Dictionary. Then we simply take the first element of the list (because your response was a list, idk why) and take the acessToken with ['accessToken'].
I have a list of LinkedIn posts IDs. I need to request share statistics for each of those posts with another request.
The request function looks like this:
def ugcp_stats(headers):
response = requests.get(f'https://api.linkedin.com/v2/organizationalEntityShareStatistics?q=organizationalEntity&organizationalEntity=urn%3Ali%3Aorganization%3A77487&ugcPosts=List(urn%3Ali%3AugcPost%3A{shid},urn%3Ali%3AugcPost%3A{shid2},...,urn%3Ali%3AugcPost%3A{shidx})', headers = headers)
ugcp_stats = response.json()
return ugcp_stats
urn%3Ali%3AugcPost%3A{shid},urn%3Ali%3AugcPost%3A{shid2},...,urn%3Ali%3AugcPost%3A{shidx} - these are the share urns. Their number depends on number of elements in my list.
What should I do next? Should I count the number of elements in my list and somehow amend the request URL to include all of them? Or maybe I should loop through the list and make a separate request for each of the elements and then append all the responses in one json file?
I'm struggling and I'm not quite sure how to write this. I don't even know how to parse the element into the request. Although I suspect it could look something like this:
for shid in shids:
def ugcp_stats(headers):
response = requests.get(f'https://api.linkedin.com/v2/organizationalEntityShareStatistics?q=organizationalEntity&organizationalEntity=urn%3Ali%3Aorganization%3A77487&ugcPosts=List(urn%3Ali%3AugcPost%3A & {shid})', headers = headers)
ugcp_stats = response.json()
return ugcp_stats
UPDATE - following your ansers
The code looks like this now:
link = "https://api.linkedin.com/v2/organizationalEntityShareStatistics?q=organizationalEntity&organizationalEntity=urn%3Ali%3Aorganization%3A77487&ugcPosts=List"
def share_stats(headers, shids):
# Local variable
sample = ""
# Sample the shids in the right pattern
for shid in shids: sample += "urn%3Ali%3AugcPost%3A & {},".format(shid)
# Get the execution of the string content
response = eval(f"requests.get('{link}({sample[:-1]})', headers = {headers})")
# Return the stats
return response.json()
if __name__ == '__main__':
credentials = 'credentials.json'
access_token = auth(credentials) # Authenticate the API
headers = headers(access_token) # Make the headers to attach to the API call.
share_stats = share_stats(headers) # Get shares
print(share_stats)
But nothing seems to be happening. It finishes the script, but I don't get anything. What's wrong?
This is just a proof of what I told you earlier as comment. Now you will adapt to your needs (even I try it to do it for you) :)
Updated - Base on your feedback.
#// IMPORT
#// I'm assuming your are using "requests" library
#// PyCharm IDE show me like this library is not used, but "eval()" is using it
import requests
#// GLOBAL VARIABLES
link: str = "https://api.linkedin.com/v2/organizationalEntityShareStatistics?q=organizationalEntity&organizationalEntity=urn%3Ali%3Aorganization%3A77487&ugcPosts=List"
#// Your function logic updated
def share_stats(sheds: list, header: dict) -> any:
# Local variable
sample = ""
# Sample the sheds in the right pattern
for shed in sheds: sample += "urn%3Ali%3AugcPost%3A & {},".format(shed)
# Get the execution of the string content
response = eval(f"requests.get('{link}({sample[:-1]})', headers = {header})")
# Return the stats as JSON file
return response.json()
#// Run if this is tha main file
if __name__ == '__main__':
#// An empty sheds list for code validation
debug_sheds: list = []
credentials: str = "credentials.json"
#// For me I get an unresolved reference for "auth", for you shod be fine
#// I'm assuming is your function for reading the file content and convert it to Python
access_token = auth(credentials) # Authenticate the API
#// Your error was from this script line
#// Error message: 'TypedDict' object is not callable
#// Your code: headers = headers(access_token)
#// When you want to get a dictionary value by a key use square brackets
headers = headers[access_token] # Make the headers to attach to the API call.
#// Here you shood ged an error/warning because you do not provided the sheds first time
#// Your code: share_stats = share_stats(headers)
share_stats = share_stats(debug_sheds, headers) # Get shares
print(share_stats)
Working on a API project, in which I'm trying to get all the redirect urls from an API output such as https://urlscan.io/api/v1/result/39a4fc22-39df-4fd5-ba13-21a91ca9a07d/
Example of where I'm trying to pull the urls from:
"redirectResponse": {
"url": "https://www.coke.com/"
I currently have the following code:
import requests
import json
import time
#URL to be scanned
url = 'https://www.coke.com'
#URL Scan Headers
headers = {'API-Key':apikey,'Content-Type':'application/json'}
data = {"url":url, "visibility": "public"}
response = requests.post('https://urlscan.io/api/v1/scan/',headers=headers, data=json.dumps(data))
uuid = response.json()['uuid']
responseUrl = response.json()['api']
time.sleep(10)
req = requests.Session()
r = req.get(responseUrl).json()
r.keys()
for value in r['data']['requests']['redirectResponse']['url']:
print(f"{value}")
I get the following error: TypeError: list indices must be integers or slices, not str. Not sure what the best way to parse the nested json in order to get all the redirect urls.
A redirectResponse isn't always present in the requests, so the code has to be written to handle that and keep going. In Python that's usually done with a try/except:
for obj in r['data']['requests']:
try:
redirectResponse = obj['request']['redirectResponse']
except KeyError:
continue # Ignore and skip to next one.
url = redirectResponse['url']
print(f'{url=!r}')
I'm very new to coding, and I'm building my first web application using open REST api with python flask.
I think the api is returning jsonp which looks like this - callbackfunction{ json }; and I get from other posts that all I need to do is getting rid of this padding. However, I can't figure out at which point I should implement the stripping.
This is my code. 5th line is throwing an error "the JSON object must be str, bytes or bytearray, not HTTPResponse"
def lookup(title):
try:
url = "http://www.aladin.co.kr/ttb/api/ItemSearch.aspx?ttbkey=foo&Query=bar"
result = urllib.request.urlopen(url)
data = json.loads(result)
data_json = data.split("{", 1)[1].strip("}")
return data_json
except requests.RequestException:
return None
I'm sure it's working well until 4th line. When I tried the code below, at least it returned result, though cryptic, like this.
b'{ "version" : "20070901", "title" :
"\xec\x95\x8c\xeb\x9d\xbc\xeb\x94\x98 \xea\xb2\x80 ...
"customerReviewRank":9 } ] };'
Judging by the keys, I'm pretty sure this is the information I requested. So what can I do to fix this? Thanks in advance!
def lookup(title):
try:
url = "http://www.aladin.co.kr/ttb/api/ItemSearch.aspx?ttbkey=foo&Query=bar"
result = urllib.request.urlopen(url)
res = result.readline()
return res