I'm trying to GET an URL of the following format using requests.get() in python:
http://api.example.com/export/?format=json&key=site:dummy+type:example+group:wheel
#!/usr/local/bin/python
import requests
print(requests.__versiom__)
url = 'http://api.example.com/export/'
payload = {'format': 'json', 'key': 'site:dummy+type:example+group:wheel'}
r = requests.get(url, params=payload)
print(r.url)
However, the URL gets percent encoded and I don't get the expected response.
2.2.1
http://api.example.com/export/?key=site%3Adummy%2Btype%3Aexample%2Bgroup%3Awheel&format=json
This works if I pass the URL directly:
url = http://api.example.com/export/?format=json&key=site:dummy+type:example+group:wheel
r = requests.get(url)
Is there some way to pass the the parameters in their original form - without percent encoding?
Thanks!
It is not good solution but you can use directly string:
r = requests.get(url, params='format=json&key=site:dummy+type:example+group:wheel')
BTW:
Code which convert payload to this string
payload = {
'format': 'json',
'key': 'site:dummy+type:example+group:wheel'
}
payload_str = "&".join("%s=%s" % (k,v) for k,v in payload.items())
# 'format=json&key=site:dummy+type:example+group:wheel'
r = requests.get(url, params=payload_str)
EDIT (2020):
You can also use urllib.parse.urlencode(...) with parameter safe=':+' to create string without converting chars :+ .
As I know requests also use urllib.parse.urlencode(...) for this but without safe=.
import requests
import urllib.parse
payload = {
'format': 'json',
'key': 'site:dummy+type:example+group:wheel'
}
payload_str = urllib.parse.urlencode(payload, safe=':+')
# 'format=json&key=site:dummy+type:example+group:wheel'
url = 'https://httpbin.org/get'
r = requests.get(url, params=payload_str)
print(r.text)
I used page https://httpbin.org/get to test it.
In case someone else comes across this in the future, you can subclass requests.Session, override the send method, and alter the raw url, to fix percent encodings and the like.
Corrections to the below are welcome.
import requests, urllib
class NoQuotedCommasSession(requests.Session):
def send(self, *a, **kw):
# a[0] is prepared request
a[0].url = a[0].url.replace(urllib.parse.quote(","), ",")
return requests.Session.send(self, *a, **kw)
s = NoQuotedCommasSession()
s.get("http://somesite.com/an,url,with,commas,that,won't,be,encoded.")
The solution, as designed, is to pass the URL directly.
The answers above didn't work for me.
I was trying to do a get request where the parameter contained a pipe, but python requests would also percent encode the pipe. So
instead i used urlopen:
# python3
from urllib.request import urlopen
base_url = 'http://www.example.com/search?'
query = 'date_range=2017-01-01|2017-03-01'
url = base_url + query
response = urlopen(url)
data = response.read()
# response data valid
print(response.url)
# output: 'http://www.example.com/search?date_range=2017-01-01|2017-03-01'
All above solutions don't seem to work anymore from requests version 2.26 on. The suggested solution from the GitHub repo seems to be using a work around with a PreparedRequest.
The following worked for me. Make sure the URL is resolvable, so don't use 'this-is-not-a-domain.com'.
import requests
base_url = 'https://www.example.com/search'
query = '?format=json&key=site:dummy+type:example+group:wheel'
s = requests.Session()
req = requests.Request('GET', base_url)
p = req.prepare()
p.url += query
resp = s.send(p)
print(resp.request.url)
Source: https://github.com/psf/requests/issues/5964#issuecomment-949013046
Please have a look at the 1st option in this github link. You can ignore the urlibpart which means prep.url = url instead of prep.url = url + qry
Related
I'm trying to GET an URL of the following format using requests.get() in python:
http://api.example.com/export/?format=json&key=site:dummy+type:example+group:wheel
#!/usr/local/bin/python
import requests
print(requests.__versiom__)
url = 'http://api.example.com/export/'
payload = {'format': 'json', 'key': 'site:dummy+type:example+group:wheel'}
r = requests.get(url, params=payload)
print(r.url)
However, the URL gets percent encoded and I don't get the expected response.
2.2.1
http://api.example.com/export/?key=site%3Adummy%2Btype%3Aexample%2Bgroup%3Awheel&format=json
This works if I pass the URL directly:
url = http://api.example.com/export/?format=json&key=site:dummy+type:example+group:wheel
r = requests.get(url)
Is there some way to pass the the parameters in their original form - without percent encoding?
Thanks!
It is not good solution but you can use directly string:
r = requests.get(url, params='format=json&key=site:dummy+type:example+group:wheel')
BTW:
Code which convert payload to this string
payload = {
'format': 'json',
'key': 'site:dummy+type:example+group:wheel'
}
payload_str = "&".join("%s=%s" % (k,v) for k,v in payload.items())
# 'format=json&key=site:dummy+type:example+group:wheel'
r = requests.get(url, params=payload_str)
EDIT (2020):
You can also use urllib.parse.urlencode(...) with parameter safe=':+' to create string without converting chars :+ .
As I know requests also use urllib.parse.urlencode(...) for this but without safe=.
import requests
import urllib.parse
payload = {
'format': 'json',
'key': 'site:dummy+type:example+group:wheel'
}
payload_str = urllib.parse.urlencode(payload, safe=':+')
# 'format=json&key=site:dummy+type:example+group:wheel'
url = 'https://httpbin.org/get'
r = requests.get(url, params=payload_str)
print(r.text)
I used page https://httpbin.org/get to test it.
In case someone else comes across this in the future, you can subclass requests.Session, override the send method, and alter the raw url, to fix percent encodings and the like.
Corrections to the below are welcome.
import requests, urllib
class NoQuotedCommasSession(requests.Session):
def send(self, *a, **kw):
# a[0] is prepared request
a[0].url = a[0].url.replace(urllib.parse.quote(","), ",")
return requests.Session.send(self, *a, **kw)
s = NoQuotedCommasSession()
s.get("http://somesite.com/an,url,with,commas,that,won't,be,encoded.")
The solution, as designed, is to pass the URL directly.
The answers above didn't work for me.
I was trying to do a get request where the parameter contained a pipe, but python requests would also percent encode the pipe. So
instead i used urlopen:
# python3
from urllib.request import urlopen
base_url = 'http://www.example.com/search?'
query = 'date_range=2017-01-01|2017-03-01'
url = base_url + query
response = urlopen(url)
data = response.read()
# response data valid
print(response.url)
# output: 'http://www.example.com/search?date_range=2017-01-01|2017-03-01'
All above solutions don't seem to work anymore from requests version 2.26 on. The suggested solution from the GitHub repo seems to be using a work around with a PreparedRequest.
The following worked for me. Make sure the URL is resolvable, so don't use 'this-is-not-a-domain.com'.
import requests
base_url = 'https://www.example.com/search'
query = '?format=json&key=site:dummy+type:example+group:wheel'
s = requests.Session()
req = requests.Request('GET', base_url)
p = req.prepare()
p.url += query
resp = s.send(p)
print(resp.request.url)
Source: https://github.com/psf/requests/issues/5964#issuecomment-949013046
Please have a look at the 1st option in this github link. You can ignore the urlibpart which means prep.url = url instead of prep.url = url + qry
I try to acess data from an Exchange Platform with API.
I have my API_Key and my SECRET_KEY, and I have the documentation of that Platform here:
https://apidocs.exir.io/
I generate the "signature-key" just as described in the documentation under "Authentication" chapter and then try to test it with a sample GET request with only one parameter.
Now if I run the Code I get "message": "Access Denied: Invalid API Signature"
Can you please help me to find the wrong thing in this code?
I think I do something wrong with params because if I use other GET orders without parameters it works!
Thank you in advance!
import time
import json
import hmac
import hashlib
import requests
API_KEY = '*****'
SECRET_KEY = '*****'
BASE_URL = 'https://api.exir.io'
timestamp = str(int(time.time()+10))
headers = {
'api-key': API_KEY,
'api-expires': timestamp} # see documentation under "Authentication"
PATH = '/v1/user/orders' # This ist just a simple example, which uses "params". See Exir documentation under "Get All Orders"
params = {"symbol":"btc-irt"}
string = 'GET'+timestamp+str(params) # see Exir API doumentation under "Authentication"
headers['api-signature'] = hmac.new(SECRET_KEY.encode('utf-8'), string.encode('utf-8'), hashlib.sha256).hexdigest()
url = 'https://api.exir.io/v1/user/orders?symbol=btc-irt'
r = requests.get(url, headers=headers)
data = r.json()
print(json.dumps(data, indent=2))
A lot of what you are doing is unnecessary, actually. The requests library handles most of what you are trying to do.
import requests
API_KEY = '*****'
SECRET_KEY = '*****'
BASE_URL = 'https://api.exir.io'
timestamp = str(int(time.time()+10))
headers = {
'api-key': API_KEY,
'api-expires': timestamp
}
url = 'https://api.exir.io/v1/user/orders?symbol=btc-irt'
r = requests.get(url, headers=headers)
The library will do the encoding for you.
You mixed the post and get request parameters. For the get request, you only need to include the params in the URL to sign. In your case it will be:
PATH = '/v1/user/orders?symbol=btc-irt'
string = 'GET/' + PATH + timestamp
I am trying to make a POST request in Python 2, using urllib2. My code is currently as follows;
url = 'http://' + server_url + '/playlists/upload?'
data = urllib.urlencode(OrderedDict([("sectionID", section_id), ("path", current_playlist), ("X-Plex-Token", plex_token)]))
req = urllib2.Request(url, data)
response = urllib2.urlopen(req)
d = response.read()
print(d)
'url' and 'data' return correctly formatted with the variables, I know this because I can copy their output into Postman for checking and the POST works fine (see example url below)
http://192.168.1.96:32400/playlists/upload?sectionID=11&path=D%3A%5CMedia%5CPPP%5Ctmp%5Cplex%5CAmbient.m3u&X-Plex-Token=XXXXXXXXX
When I run my Python code I get a 401 error returned, presumably meaning the X-Plex-Token parameter was not correctly sent, hence I am not allowed access.
Can anyone tell me where I'm going wrong? Help is greatly appreciated.
Have you tried removing the question mark and not using OrderedDict (no idea why you would need that) ?
url = 'http://' + server_url + '/playlists/upload'
data = urllib.urlencode({"sectionID":section_id), "path":current_playlist,"X-Plex-Token":plex_token})
req = urllib2.Request(url, data)
response = urllib2.urlopen(req)
d = response.read()
print(d)
Of course you should be using requests instead anyway:
import requests
r = requests.post('http://{}/playlists/upload'.format(server_url), data = {"sectionID":section_id), "path":current_playlist,"X-Plex-Token":plex_token})
print r.url
print r.text
print r.json
I've ended up switching to Python 3, as I didn't realise that the requests module was included by default. Still no idea why the above wasn't working, but maybe something to do with the lack of headers
headers = {'cache-control': "no-cache"}
edit:
This is what I'm using now, as mentioned above I probably don't need OrderedDict.
import requests
url = 'http://' + server_url + '/playlists/upload'
headers = {'cache-control': "no-cache"}
querystring = urllib.parse.urlencode(OrderedDict([("sectionID", section_id), ("path", current_playlist), ("X-Plex-Token", plex_token)]))
response = requests.request("POST", url, data = "", headers = headers, params = querystring)
print(response.text)
I'm trying to use the News API in a python program, and for some reason I can't get a 200 response no matter what. I'm pretty unfamiliar with the requests library, so maybe I'm not doing something right, but here's what my code looks like:
api = XXXXXXXXXX
def get_json_response(apiKey, resource='google-news', sortBy='latest'):
url = 'https://newsapi.org/v1/articles'
headers = { 'source': resource,
'apiKey': apiKey,
'sortBy': sortBy}
r = requests.get(url, headers=headers)
print(r.status_code)
get_json_response(api)
and the output is always 401.
But what's weird is if i just put "https://newsapi.org/v1/articles/?source=google-news&apiKey=XXXXXXXXX" in a browser, it gives the correct json response, so it has to be something wrong with how I'm using requests.
Any ideas? Thanks in advance
EDIT:
Not exactly an elegant solution, but i switched the line to:
r = requests.get(url + '/?source=' + resource + '&sortBy=' + sortBy + '&apiKey=' + apiKey)
And that worked, but I'd still like to know how to use the requests package correctly for the future.
Based on the 'working' link provided, it expects URL parameters, not headers on its request, so:
def get_json_response(apiKey, resource='google-news'):
url = 'https://newsapi.org/v1/articles/'
params = {'source': resource,
'apiKey': apiKey}
r = requests.get(url, params=params)
print(r.status_code)
# etc.
I have an existing Http POST using urllib2:
data = 'client_id=%s&client_secret=%s&grant_type=authorization_code&code=%s&redirect_uri=%s' % (settings.GOOGLE_CLIENT_ID, settings.GOOGLE_CLIENT_SECRET, code, redirect_uri)
req = urllib2.Request(access_token_url, data=data)
response = urllib2.urlopen(req)
response_content = response.read()
json_response = json.loads(response_content)
I'm trying to convert this to the Requests library instead (http://docs.python-requests.org/) but I'm getting a 400 Invalid Request.
Here's my attempt:
params = {'redirect_uri' : redirect_uri}
params['client_id'] = settings.GOOGLE_CLIENT_ID
params['client_secret'] = settings.GOOGLE_CLIENT_SECRET
params['grant_type'] = 'authorization_code'
params['code'] = code
req = requests.post(access_token_url, data=params)
json_response = req.json()
I tried tweaking it to use params instead of data but I got the same error.
Anything I'm missing?
Make sure the values of the data dict are not already escaped as requests will do that for you. Please notice how your original example does not do any escaping.