Develop Reference

Why doesn't the json() method on the requests module return anything?

When I call the json() method on request response I get an error.
Any suggestions to what could be wrong here?
My code:
import requests
import bs4
url = 'https://www.reddit.com/r/AskReddit/comments/l4styp/serious_what_is_the_the_scariest_thing_that_you/'
rsp = requests.get(url)
sc = rsp.json()
print(sc)
Output:
File "c:\VS_Code1\scrape.py", line 6, in <module>
sc = rsp.json()
File "C:\Users\User\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\requests\models.py", line 900, in json
return complexjson.loads(self.text, **kwargs)
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.496.0_x64__qbz5n2kfra8p0\lib\json\__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.496.0_x64__qbz5n2kfra8p0\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.496.0_x64__qbz5n2kfra8p0\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 2 column 5 (char 5)

Your rsp is actually returning <Response [200]> which is not a JSON. If you want to read the content of the response, you can simply do:
rsp.text
What you get from the URL you posted here is HTML, and not JSON.

This does not work because the page you are fetching does not return Json but HTML source code instead.
To fetch the content of the webpage you need to replace sc = rsp.json() with sc = rsp.text
If you need this data in Json, you can look into Reddit's API: https://www.reddit.com/dev/api

I figured out that if I want to get the JSON from the URL that I've inputted I have to add .json to the end of the URL, this is for some reason (to my knowledge) unique to Reddit and a few other sites that allow it.

API returned JSON data wrapped in a function, breaking json.loads

I'm trying to work with JSON data that is pulled from USGS Earthquake API. If you follow that link, you can see the raw JSON data.
The JSON looks great; however, the returned request is wrapped in an eqfeed_callback(); that is breaking the JSON deserializer in Python.
A quick look at the code I have so far:
import requests
import json
from pprint import pprint
URL = "http://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/2.5_week.geojsonp"
response = requests.get(URL)
raw_json = str(response.content)
json = json.loads(raw_json)
print(json)
I get the errors:
Traceback (most recent call last):
File "run.py", line 11, in <module>
json = json.loads(raw_json)
File "C:\Program Files\Anaconda3\lib\json\__init__.py", line 354, in loads
return _default_decoder.decode(s)
File "C:\Program Files\Anaconda3\lib\json\decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Program Files\Anaconda3\lib\json\decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Although I'm positive the issue is that it's wrapped in that function and the JSON decoder doesn't like it. So how would I go about removing the function wrapper to leave me with the clean JSON inside.

You're using the wrong URL.
JSON wrapped in a function call is JSONP, which is needed for getting around CORS when calling an API from web browsers.
The URL to get normal JSON is
URL = "http://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/2.5_week.geojson"

Can not parse response from sg.media-imdb in python

I'm trying to parse response from https://sg.media-imdb.com/suggests/a/a.json in Python 3.6.8.
Here is my code:
import requests
url = 'https://sg.media-imdb.com/suggests/a/a.json'
data = requests.get(url).json()
I get this error:
$ /usr/bin/python3 /home/livw/Python/test_scrapy/phase_1.py
Traceback (most recent call last):
File "/home/livw/Python/test_scrapy/phase_1.py", line 33, in <module>
data = requests.get(url).json()
File "/home/livw/.local/lib/python3.6/site-packages/requests/models.py", line 889, in json
self.content.decode(encoding), **kwargs
File "/usr/lib/python3/dist-packages/simplejson/__init__.py", line 518, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3/dist-packages/simplejson/decoder.py", line 370, in decode
obj, end = self.raw_decode(s)
File "/usr/lib/python3/dist-packages/simplejson/decoder.py", line 400, in raw_decode
return self.scan_once(s, idx=_w(s, idx).end())
simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
It seems like the response format is not JSON format, although I can parse the response at JSON Formatter & Validator
How to fix it and store the response in a json object?

This probably happend because its not a complete json, it have a prefix
you can see that the response start with imdb$a( and ends with )
json parsing doesn't know how to handle it and he fails, you can remove those values and just parse the json itself
you can do this:
import json
import requests
url = 'https://sg.media-imdb.com/suggests/a/a.json'
data = requests.get(url).text
json.loads(data[data.index('{'):-1])

JSONDecodeError Expecting value

I am trying to get json object, and it tells me it's expecting a value even though i define the path to the json in r.json(). Also when i do r.headers[content-type] give me text/html;charset=ISO-8859-1 ... Thank you for your time everyone
import requests
import json
session = requests.Session()
username = "------"
password = "-------"
url_cookie = 'http://ludwig.podiumdata.com:----/podium/j_spring_security_check?j_username=--&j_password=----'
url_get = 'http://ludwig.corp.podiumdata.com:----/qdc/entity/v1/getEntities?type=EXTERNAL&count=2&sortAttr=name&sortDir=ASC'
r = requests.get(url_get, auth=(username,password), verify=False)
r.json()
r.headers['content-type']
Traceback (most recent call last):
File "<ipython-input-108-61f8159bb1b5>", line 10, in <module>
r.json()
File "//anaconda3/lib/python3.7/site-packages/requests/models.py", line 897, in json
return complexjson.loads(self.text, **kwargs)
File "//anaconda3/lib/python3.7/json/__init__.py", line 348, in loads
return _default_decoder.decode(s)
File "//anaconda3/lib/python3.7/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "//anaconda3/lib/python3.7/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
JSONDecodeError: Expecting value

Your request is receiving the response as text/html, but you want to receive application/json.
You need to include {'Accept': 'application/json'} as a header in your request.
Example: requests.get(url_get, auth=(username,password), headers={'Accept': 'application/json'}, verify=False)
Also, it looks like you aren't using the Session you created on line 3, but that isn't what is causing this error.

AttributeError : object has no attribute

I got a program from a formerly colleague and now should maintain it.
This python script asks our Jira instance with a given jql ( on the API ).
The return is a list of all issues, which are matching the search criteria.
But now it's not working, and I receive on the server ( Ubuntu ) and on my local windows PC a Json error message.
note : it ran for about a year not, but back then it worked.
Here is what the script looks like :
import json
import subprocess
jiraSerachUrl = "https://ourJiraInstance.net/rest/api/2/search?jql=key%20=%20%22TEST-123%22"
jiraResponse = subprocess.Popen(["curl","-l","-s","-u", "jiraUser"+":"+"jiraUserPassword", "-X", "GET", jiraSerachUrl ],stdout=subprocess.PIPE,shell=True).communicate()[0]
## shell=True only added for Windows Instance
print(type(jiraResponse))
##print = <class 'bytes'>
print(jiraResponse)
## print = b''
jiraJsonResponse = json.loads(jiraResponse.decode('utf-8'))
print(jiraJsonResponse)
The jql/jira search address returns the following (shorted answer, all fields of the task are returned):
{"expand":"names,schema","startAt":0,"maxResults":50,"total":1,"issues":
[{"expand":"operations,versionedRepresentations,editmeta,changelog,transitions,renderedFields",
"id":"145936","self":"https://ourJiraInstance.net/rest/api/2/issue/145936","key":"TEST-123","fields":{"parent": ...
The Error on the Windows PC is the following
Traceback (most recent call last): File
"C:\Users\User\Desktop\test.py", line 10, in
jiraJsonResponse = json.loads(jiraResponse.decode('utf-8')) File "C:\Users\User\AppData\Local\Programs\Python\Python35-32\lib\json__init__.py",
line 319, in loads
return _default_decoder.decode(s) File "C:\Users\User\AppData\Local\Programs\Python\Python35-32\lib\json\decoder.py",
line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "C:\Users\User\AppData\Local\Programs\Python\Python35-32\lib\json\decoder.py",
line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char
0)
This is the error on the Ubuntu Server ( running the same script )
Traceback (most recent call last): File "searchJira.py", line 33, in
jiraJsonResponse = json.loads(jiraResponse) File "/usr/lib/python2.7/json/init.py", line 338, in loads
return _default_decoder.decode(s) File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode
raise ValueError("No JSON object could be decoded") ValueError: No JSON object could be decoded
So far I tried to change the Json load to simpleJson, but with the same result.
Changing the format to which Json should decode ( e.g. unicode ) took no effect.

I have tried a bit and finaly got it. by replacing curl with responses i got finally the result I wanted. my request looks now like this :
r = requests.get(jiraSerachUrl,auth=HTTPBasicAuth(user, password), verify=False)
jiraJsonResponse=json.loads(r.text)