Hello there,
I get the following error when running the following script in Python:
import requests
r = requests.get('https://www.instagram.com/p/CJDxE7Yp5Oj/?__a=1')
data = r.json()['graphql']['shortcode_media']
C:\ProgramData\Anaconda3\envs\test\python.exe C:/Users/Solba/PycharmProjects/test/main.py
Traceback (most recent call last):
File "C:/Users/Solba/PycharmProjects/test/main.py", line 4, in
data = r.json()
File "C:\ProgramData\Anaconda3\envs\test\lib\site-packages\requests\models.py", line 900, in json
return complexjson.loads(self.text, **kwargs)
File "C:\ProgramData\Anaconda3\envs\test\lib\json_init_.py", line 357, in loads
return _default_decoder.decode(s)
File "C:\ProgramData\Anaconda3\envs\test\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\ProgramData\Anaconda3\envs\test\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Process finished with exit code 1
Python version: 3.9
PyCharm version: 2020.3.1
Anaconda version: 1.10.0
Please help. Thank u.
r.json() expects a JSON string to be returned by the API. The API should explicitly say it is responding with JSON through response headers.
In this case, the URL you are requesting is either not responding with a proper JSON or not explicitly saying it is responding with a JSON.
You can first check the response sent by the URL by:
data = r.text
print(data)
If the response can be treated as a JSON string, then you can process it with:
import json
data = json.loads(r.text)
Note:
You can also check the content-type and Accept headers to ensure the request and response are in the required datatype
The reason is because the response is not returning JSON, but instead a whole HTML page. Try r.text instead of r.json()..., and then do whatever you want from there.
If you are not sure the type of content it returns:
h = requests.head('https://www.instagram.com/p/CJDxE7Yp5Oj/?__a=1')
header = h.headers
contentType = header.get('content-type')
print(contentType)
Based on your URL, it returns text/html.
Alternatively, you can try to add a User-Agent in your request - this is to emulate the request to make it look like it comes from a browser, and not a script.
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/46.0.2490.80'
}
r = requests.get('https://www.instagram.com/p/CJDxE7Yp5Oj/?__a=1', headers=headers)
data = r.json()
Related
I am making a Python request using the request library to TikTok. I managed to dig up the URL for their user details (I don't know whether this is legal or not. If it isn't, please let me know). When I try to parse it into json, it raises an exception. Could someone helpme parse/fix this? Here is the code:
Python Code:
r1 = requests.get("https://www.tiktok.com/node/share/user/#nike?isUniqueId=true&verifyFp=verify_kb51zknj_GH98fcme_eDuR_4XzM_ATwp_s8TRdCzr8fwi&_signature=KBbp4AAgEBCtR.e4r-y0ZSgWqPAAHbR").json()
print(r1)
Output:
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/requests/models.py", line 897, in json
return complexjson.loads(self.text, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/__init__.py", line 348, in loads
return _default_decoder.decode(s)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Thanks
you must provide user-agent header, ie
headers = {
'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36',
}
r1 = requests.get("https://www.tiktok.com/node/share/user/#nike?isUniqueId=true&verifyFp=verify_kb51zknj_GH98fcme_eDuR_4XzM_ATwp_s8TRdCzr8fwi&_signature=KBbp4AAgEBCtR.e4r-y0ZSgWqPAAHbR", headers=headers).json()
print(r1)
sory I tried to post it as comment, but failed to format the code xD
I wrote a wrapper in Python that allows you to fetch Users, Videos, Hashtags, Videos by music, etc.
The project can be found here - TikTokAPI-Python
For your problem of fetching a user -
Install
pip install PyTikTokAPI
Get user
from TikTokAPI import TikTokAPI
api = TikTokAPI()
user_obj = api.getUserByName("fcbarcelona")
I have written a script which is hitting Azure DevOps API and get API response from it. The script is working file on local machine but when I am trying to run the same script from Azure pipeline it is giving me error.
Traceback (most recent call last):
File "/azp/$AZP_AGENT_NAME/3/s/terra/agent/agent_queue_status.py", line 19, in <module>
data = json.loads(r)
File "/_work/_tool/Python/3.9.0/x64/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/_work/_tool/Python/3.9.0/x64/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/_work/_tool/Python/3.9.0/x64/lib/python3.9/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 3 column 1 (char 4)
##[error]Bash exited with code '1'.
This the my script:
import base64
import requests
import json
AZURE_DEVOPS_PAT = 'my_pat'
authorization = str(base64.b64encode(bytes(':' + AZURE_DEVOPS_PAT, 'ascii')), 'ascii')
headers = {
'Accept': 'application/json',
'Authorization': 'Basic ' + authorization
}
url = 'https://dev.azure.com/{MY-ORG}/(MY-Project)/_settings/agentqueues?__rt=fps&__ver=2'
r = requests.get(url, headers=headers).text
data = json.loads(r)
print(data)
I tried everything. Like changing the way of getting response but no luck.
tried:-
r = requests.get(url, headers=headers).json
-------------------------------------------
r = requests.get(url, headers=headers)
data = r.json()
Please suggest something. The code is working fine on local machine but not with pipeline on Azure agent.
I am pretty new to python and learning how to make HTTP request and store the response in a variable.
Below is the similar kind of code snippet that I am trying to make the POST request.
import requests
import simplejson as json
api_url = https://jsonplaceholder.typicode.com/tickets
raw_body = {"searchBy":"city","searchValue":"1","processed":9,"size":47,"filter":{"cityCode":["BA","KE","BE"],"tickets":["BLUE"]}}
raw_header = {"X-Ticket-id": "1234567", "X-Ticket-TimeStamp": "11:01:1212", "X-Ticket-MessageId": "123", 'Content-Type': 'application/json'}
result = requests.post(api_url, headers=json.loads(raw_header), data=raw_body)
#Response Header
response_header_contentType = result.headers['Content-Type'] #---> I am getting response_header_contentType as "text/html; charset=utf-8"
#Trying to get the result in json format
response = result.json() # --> I am getting error at this line. May be because the server is sending the content type as "text/html" and I am trying to capture the json response.
Error in console :
Traceback (most recent call last):
File "C:\sam\python-project\v4\makeApiRequest.py", line 45, in make_API_request
response = result.json()
File "C:\Users\userName\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\models.py", line 898, in json
return complexjson.loads(self.text, **kwargs)
File "C:\Users\userName\AppData\Local\Programs\Python\Python37\lib\site-packages\simplejson\__init__.py", line 525, in loads
return _default_decoder.decode(s)
File "C:\Users\userName\AppData\Local\Programs\Python\Python37\lib\site-packages\simplejson\decoder.py", line 370, in decode
obj, end = self.raw_decode(s)
File "C:\Users\userName\AppData\Local\Programs\Python\Python37\lib\site-packages\simplejson\decoder.py", line 400, in raw_decode
return self.scan_once(s, idx=_w(s, idx).end())
simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
So, how can I store the response in a variable based on the content-type sent by the server using requests.
Can somebody please help me here. I tried googling too but did not find any helpful documentation on how to capture the response based on the content-type.
as you already said your contentType is 'text/html' not 'application/json' that normally means that it can not be decoded as json.
If you look at the documentation
https://2.python-requests.org/en/master/user/quickstart/#response-content you can find that there are different ways to decode the body, if you already know you have 'text/html' it makes sense to decode it with response.text.
Hence it makes sense to distinquish based on the content type how to decode your data:
if result.headers['Content-Type'] == 'application/json':
data = result.json()
elif result.headers['Content-Type'] == 'text/html':
data = result.text
else:
data = result.raw
I am trying to get json object, and it tells me it's expecting a value even though i define the path to the json in r.json(). Also when i do r.headers[content-type] give me text/html;charset=ISO-8859-1 ... Thank you for your time everyone
import requests
import json
session = requests.Session()
username = "------"
password = "-------"
url_cookie = 'http://ludwig.podiumdata.com:----/podium/j_spring_security_check?j_username=--&j_password=----'
url_get = 'http://ludwig.corp.podiumdata.com:----/qdc/entity/v1/getEntities?type=EXTERNAL&count=2&sortAttr=name&sortDir=ASC'
r = requests.get(url_get, auth=(username,password), verify=False)
r.json()
r.headers['content-type']
Traceback (most recent call last):
File "<ipython-input-108-61f8159bb1b5>", line 10, in <module>
r.json()
File "//anaconda3/lib/python3.7/site-packages/requests/models.py", line 897, in json
return complexjson.loads(self.text, **kwargs)
File "//anaconda3/lib/python3.7/json/__init__.py", line 348, in loads
return _default_decoder.decode(s)
File "//anaconda3/lib/python3.7/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "//anaconda3/lib/python3.7/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
JSONDecodeError: Expecting value
Your request is receiving the response as text/html, but you want to receive application/json.
You need to include {'Accept': 'application/json'} as a header in your request.
Example: requests.get(url_get, auth=(username,password), headers={'Accept': 'application/json'}, verify=False)
Also, it looks like you aren't using the Session you created on line 3, but that isn't what is causing this error.
Recenetly I have been doing the web spider for fun. I want to learn how to login a website with verify code. One way I learnt is to use cookies. So I had a try. But I realised a problem.
For example, I want to use request.session to get url:www.lovetvshow.com
And I can get all the html text, but when I was trying to convert it to Json, it failed. It always shows "[ValueError] No JSON object could be decoded". But I have already had the text. Why is it no json object?
session = requests.session()
login_data = {'email': email, 'password': password}
header = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36',
'Host': 'www.lovetvshow.com'
}
# r = session.post('http://www.renren.com/', data=login_data, headers=header)
r = session.get('http://www.lovetvshow.com/',headers=header)
print r
print r.json()
This will yield:
<Response [200]>
Traceback (most recent call last):
File "C:/Users/Hao/PycharmProjects/WebSpiderTutorial1/WebSpiderTutorial1.py", line 128, in <module>
requests_session, requests_cookies = create_session()
File "C:/Users/Hao/PycharmProjects/WebSpiderTutorial1/WebSpiderTutorial1.py", line 104, in create_session
print r.json()
File "C:\Python27\lib\site-packages\requests\models.py", line 892, in json
return complexjson.loads(self.text, **kwargs)
File "C:\Python27\lib\json\__init__.py", line 338, in loads
return _default_decoder.decode(s)
File "C:\Python27\lib\json\decoder.py", line 365, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Python27\lib\json\decoder.py", line 383, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
Any suggestions? Thanks a head.
You have to make sure there is JSON to be decoded. Check the complete text using
print print r.text . And look for your json