I have the following code using urllib in Python 2.7 and its working. I'm trying to do the same request using the requests library but I cant get it to work.
import urllib
import urllib2
import json
req = urllib2.Request(url='https://testone.limequery.com/index.php/admin/remotecontrol',\
data='{\"method\":\"get_session_key\",\"params\":[\"username\",\"password\"],\"id\":1}')
req.add_header('content-type', 'application/json')
req.add_header('connection', 'Keep-Alive')
f = urllib2.urlopen(req)
myretun = f.read()
j=json.loads(myretun)
print(j['result'])
Using requests library( Doesn't work)
import requests
import json
d= {"method":"get_session_key","params":["username","password"],"id":"1"}
headers = {'content-type' :'application/json','connection': 'Keep-Alive'}
req2 = requests.get(url='https://testone.limequery.com/index.php/admin/remotecontrol',data=d,headers=headers)
json_data = json.loads(req2.text)
print(json data['result'])
I'm getting an error JSONDecodeError: Expecting value: line 1 column 1 (char 0) How can I make the code work with the requests library?
First, you're sending the wrong type of request. You're sending a GET request, but you need to send a POST, with requests.post.
Second, passing a dict as data will form-encode the data rather than JSON-encoding it. If you want to use JSON in your request body, use the json argument, not data:
requests.post(url=..., json=d)
Reference Link: http://docs.python-requests.org/en/master/api/
You can use requests module of python like so
import requests
Req = requests.request(
method = "GET", # or "POST", "PUT", "DELETE", "PATCH" etcetera
url = "http(s)://*",
params = {"key": "value"}, # IF GET Request (Optional)
data = {"key": "value"}, # IF POST Request (Optional)
headers = {"header_name": "header_value"} # (Optional)
)
print Req.content
You can surround the code with try::catch block like below to catch any exception thrown by requests module
try:
# requests.request(** Arguments)
except requests.exceptions.RequestException as e:
print e
For full argument list, please check reference link.
Related
I need [0] to increase everytime and fetch the data when index change. from 0 to 13
import requests as r
import json
url = "https://services6.arcgis.com/bKYAIlQgwHslVRaK/arcgis/rest/services/CasesByRegion_ViewLayer/FeatureServer/0/query?where=1%3D1&outFields=*&outSR=4326&f=json"
response = urlopen(url)
Data= json.load(response )
for index in Data:
list = Data['features'][0]['attributes']
[0]+1
print(list)
Here is another simple approach without using urllib:
import requests as r
import json
url = "https://jsonplaceholder.typicode.com/todos/1"
response = r.get(url)
data = response.json()
print(data)
requests.get().json() delivers the complete dict from the response payload:
import requests as r
response = r.get(url)
Data = response.json()
Your json.load() doesn't work as expected because response is a dictionary from the requests module, containing some HTTP stuff like status code, reason, encoding. For API calls, this is not what you want (HTTP errors should be handled with exceptions). What you want is response.json() or response.text.
Also, you imported requests but didn't use it? I don't know about urlopen(). Use requests.get().
I'm trying to GET an URL of the following format using requests.get() in python:
http://api.example.com/export/?format=json&key=site:dummy+type:example+group:wheel
#!/usr/local/bin/python
import requests
print(requests.__versiom__)
url = 'http://api.example.com/export/'
payload = {'format': 'json', 'key': 'site:dummy+type:example+group:wheel'}
r = requests.get(url, params=payload)
print(r.url)
However, the URL gets percent encoded and I don't get the expected response.
2.2.1
http://api.example.com/export/?key=site%3Adummy%2Btype%3Aexample%2Bgroup%3Awheel&format=json
This works if I pass the URL directly:
url = http://api.example.com/export/?format=json&key=site:dummy+type:example+group:wheel
r = requests.get(url)
Is there some way to pass the the parameters in their original form - without percent encoding?
Thanks!
It is not good solution but you can use directly string:
r = requests.get(url, params='format=json&key=site:dummy+type:example+group:wheel')
BTW:
Code which convert payload to this string
payload = {
'format': 'json',
'key': 'site:dummy+type:example+group:wheel'
}
payload_str = "&".join("%s=%s" % (k,v) for k,v in payload.items())
# 'format=json&key=site:dummy+type:example+group:wheel'
r = requests.get(url, params=payload_str)
EDIT (2020):
You can also use urllib.parse.urlencode(...) with parameter safe=':+' to create string without converting chars :+ .
As I know requests also use urllib.parse.urlencode(...) for this but without safe=.
import requests
import urllib.parse
payload = {
'format': 'json',
'key': 'site:dummy+type:example+group:wheel'
}
payload_str = urllib.parse.urlencode(payload, safe=':+')
# 'format=json&key=site:dummy+type:example+group:wheel'
url = 'https://httpbin.org/get'
r = requests.get(url, params=payload_str)
print(r.text)
I used page https://httpbin.org/get to test it.
In case someone else comes across this in the future, you can subclass requests.Session, override the send method, and alter the raw url, to fix percent encodings and the like.
Corrections to the below are welcome.
import requests, urllib
class NoQuotedCommasSession(requests.Session):
def send(self, *a, **kw):
# a[0] is prepared request
a[0].url = a[0].url.replace(urllib.parse.quote(","), ",")
return requests.Session.send(self, *a, **kw)
s = NoQuotedCommasSession()
s.get("http://somesite.com/an,url,with,commas,that,won't,be,encoded.")
The solution, as designed, is to pass the URL directly.
The answers above didn't work for me.
I was trying to do a get request where the parameter contained a pipe, but python requests would also percent encode the pipe. So
instead i used urlopen:
# python3
from urllib.request import urlopen
base_url = 'http://www.example.com/search?'
query = 'date_range=2017-01-01|2017-03-01'
url = base_url + query
response = urlopen(url)
data = response.read()
# response data valid
print(response.url)
# output: 'http://www.example.com/search?date_range=2017-01-01|2017-03-01'
All above solutions don't seem to work anymore from requests version 2.26 on. The suggested solution from the GitHub repo seems to be using a work around with a PreparedRequest.
The following worked for me. Make sure the URL is resolvable, so don't use 'this-is-not-a-domain.com'.
import requests
base_url = 'https://www.example.com/search'
query = '?format=json&key=site:dummy+type:example+group:wheel'
s = requests.Session()
req = requests.Request('GET', base_url)
p = req.prepare()
p.url += query
resp = s.send(p)
print(resp.request.url)
Source: https://github.com/psf/requests/issues/5964#issuecomment-949013046
Please have a look at the 1st option in this github link. You can ignore the urlibpart which means prep.url = url instead of prep.url = url + qry
I am trying to pass a list of dictionaries(strings) to a for a put request. I am getting this error:
TypeError: POST data should be bytes, an iterable of bytes.
Is this the right way to make a put request with list of dictionaries(strings) in python.
list looks like the following:
list1 = ['{"id" : "","email" : "John#fullcontact.com","fullName": "John Lorang"}', '{"id" : "","email" : "Lola#fullcontact.com","fullName": "Lola Dsilva"}']
myData = json.dumps(list1)
myRestRequestObj = urllib.request.Request(url,myData)
myRestRequestObj.add_header('Content-Type','application/json')
myRestRequestObj.add_header('Authorization','Basic %s')
myRestRequestObj.get_method = lambda : 'PUT'
try:
myRestRequestResponse = urllib.request.urlopen(myRestRequestObj)
except urllib.error.URLError as e:
print(e.reason)
As you said in a comment, you cannot use requests (that's pretty sad to hear!), so I did another snippet using urllib (the short answer: you must .encode('utf-8') json.dumps and decode('utf-8') response.read()):
import urllib.request
import urllib.error
import json
url = 'http://httpbin.org/put'
token = 'jwtToken'
list1 = ['{"id" : "","email" : "John#fullcontact.com","fullName": "John Lorang"}', '{"id" : "","email" : "Lola#fullcontact.com","fullName": "Lola Dsilva"}']
# Request needs bytes, so we have to encode it
params = json.dumps(list1).encode('utf-8')
headers = {
'Content-Type': 'application/json',
'Authorization': 'Basic {token}'.format(token=token)
}
# Let's try to create our request with data, headers and method
try:
request = urllib.request.Request(url, data=params, headers=headers, method='PUT')
except urllib.error.URLError as e:
# Unable to create our request, here the reason
print("Unable to create youro request: {error}".format(error=str(e)))
else:
# We did create our request, let's try to use it
try:
response = urllib.request.urlopen(request)
except urllib.error.HTTPError as e:
# An HTTP error occured, here the reason
print("HTTP Error: {error}".format(error=str(e)))
except Exception as e:
# We got another reason, here the reason
print("An error occured while trying to put {url}: {error}".format(
url=url,
error=str(e)
))
else:
# We are printing the result
# We must decode it because response.read() returns a bytes string
print(response.read().decode('utf-8'))
I did try to add some comments. I hope this solution help you!
To help you learn a better way to learn python, you should read the Style Guide for Python Code
I will suppose you can use the requests module (pip install requests).
requests is a simple yet powerful HTTP libraby for Python.
import json
import requests
my_data = json.dumps(list1)
headers = {
'Authorization': 'Basic {token}'.format(token=your_token)
}
response = requests.put(url, headers=headers, json=my_data)
print("Status code: {status}".format(status=response.status_code))
print("raw response: {raw_response}".format(
raw_response=response.text
)
print("json response: {json_response}".format(
json_response=response.json()
)
I'm trying to retrieve the response json data by a web site that I call.
The site is this:
WebSite DriveNow
On this page are shown on map some data. With browser debugger I can see the end point
end point
that sends response data json.
I have use this python to try scrape the json response data:
import requests
import json
headers = {
'Host': 'api2.drive-now.com',
'X-Api-Key': 'adf51226795afbc4e7575ccc124face7'
}
r = requests.get('https://api2.drive-now.com/cities/42756?expand=full', headers=headers)
json_obj = json.loads(r.content)
but I get this error:
hostname doesn't match either of 'activityharvester.com'
How I can retrieve this data?
Thanks
I have tried to call the endpoint that show json response using Postam, and passing into Header only Host and Api-Key. The result is the json that i want. But i i try the same call into python i recive the error hostname doesn't match either of 'activityharvester.com'
I don't understand your script, nor your question. Why two requests and three headers ? Did you mean something like this ?
import requests
import json
headers = {
'User-Agent': 'Mozilla/5.0',
'X-Api-Key':'adf51226795afbc4e7575ccc124face7',
}
res = requests.get('https://api2.drive-now.com/cities/4604?expand=full', headers=headers, allow_redirects=False)
print(res.status_code, res.reason)
json_obj = json.loads(res.content)
print(json_obj)
I'm trying to GET an URL of the following format using requests.get() in python:
http://api.example.com/export/?format=json&key=site:dummy+type:example+group:wheel
#!/usr/local/bin/python
import requests
print(requests.__versiom__)
url = 'http://api.example.com/export/'
payload = {'format': 'json', 'key': 'site:dummy+type:example+group:wheel'}
r = requests.get(url, params=payload)
print(r.url)
However, the URL gets percent encoded and I don't get the expected response.
2.2.1
http://api.example.com/export/?key=site%3Adummy%2Btype%3Aexample%2Bgroup%3Awheel&format=json
This works if I pass the URL directly:
url = http://api.example.com/export/?format=json&key=site:dummy+type:example+group:wheel
r = requests.get(url)
Is there some way to pass the the parameters in their original form - without percent encoding?
Thanks!
It is not good solution but you can use directly string:
r = requests.get(url, params='format=json&key=site:dummy+type:example+group:wheel')
BTW:
Code which convert payload to this string
payload = {
'format': 'json',
'key': 'site:dummy+type:example+group:wheel'
}
payload_str = "&".join("%s=%s" % (k,v) for k,v in payload.items())
# 'format=json&key=site:dummy+type:example+group:wheel'
r = requests.get(url, params=payload_str)
EDIT (2020):
You can also use urllib.parse.urlencode(...) with parameter safe=':+' to create string without converting chars :+ .
As I know requests also use urllib.parse.urlencode(...) for this but without safe=.
import requests
import urllib.parse
payload = {
'format': 'json',
'key': 'site:dummy+type:example+group:wheel'
}
payload_str = urllib.parse.urlencode(payload, safe=':+')
# 'format=json&key=site:dummy+type:example+group:wheel'
url = 'https://httpbin.org/get'
r = requests.get(url, params=payload_str)
print(r.text)
I used page https://httpbin.org/get to test it.
In case someone else comes across this in the future, you can subclass requests.Session, override the send method, and alter the raw url, to fix percent encodings and the like.
Corrections to the below are welcome.
import requests, urllib
class NoQuotedCommasSession(requests.Session):
def send(self, *a, **kw):
# a[0] is prepared request
a[0].url = a[0].url.replace(urllib.parse.quote(","), ",")
return requests.Session.send(self, *a, **kw)
s = NoQuotedCommasSession()
s.get("http://somesite.com/an,url,with,commas,that,won't,be,encoded.")
The solution, as designed, is to pass the URL directly.
The answers above didn't work for me.
I was trying to do a get request where the parameter contained a pipe, but python requests would also percent encode the pipe. So
instead i used urlopen:
# python3
from urllib.request import urlopen
base_url = 'http://www.example.com/search?'
query = 'date_range=2017-01-01|2017-03-01'
url = base_url + query
response = urlopen(url)
data = response.read()
# response data valid
print(response.url)
# output: 'http://www.example.com/search?date_range=2017-01-01|2017-03-01'
All above solutions don't seem to work anymore from requests version 2.26 on. The suggested solution from the GitHub repo seems to be using a work around with a PreparedRequest.
The following worked for me. Make sure the URL is resolvable, so don't use 'this-is-not-a-domain.com'.
import requests
base_url = 'https://www.example.com/search'
query = '?format=json&key=site:dummy+type:example+group:wheel'
s = requests.Session()
req = requests.Request('GET', base_url)
p = req.prepare()
p.url += query
resp = s.send(p)
print(resp.request.url)
Source: https://github.com/psf/requests/issues/5964#issuecomment-949013046
Please have a look at the 1st option in this github link. You can ignore the urlibpart which means prep.url = url instead of prep.url = url + qry