I'm trying to build a website using web.py, which is able to search the mobile.de database (mobile.de is a German car sales website). For this I need to use the mobile.de API and make a GET request to it doing the following (this is an example from the API docs):
GET /1.0.0/ad/search?exteriorColor=BLACK&modificationTime.min=2012-05-04T18:13:51.0Z HTTP/1.0
Host: services.mobile.de
Authorization: QWxhZGluOnNlc2FtIG9wZW4=
Accept: application/xml
(The authorization needs to be my username and password joined together using a colon and then being encoded using Base64.)
So I use urllib2 to do the request as follows:
>>> import base64
>>> import urllib2
>>> headers = {'Authorization': base64.b64encode('myusername:mypassw'), 'Accept': 'application/xml'}
>>> req = urllib2.Request('http://services.mobile.de/1.0.0/ad/search?exteriorColor=BLACK', headers=headers)
And from here I am unsure how to proceed. req appears to be an instance with some methods to get the information in it. But did it actually send the request? And if so, where can I get the response?
All tips are welcome!
You need to call req.read() to call the URL and get the response.
But you'd be better off using the requests library, which is much easier to use.
Related
I used to selenium for downloading special reports from webpage where I have to login. Webpage has integrated OKTA Authentication plugin . I find out that there would be better and more effective use internal API requests. So I tried find how to use request python library with creating session, but I am unsuccessful. I tried this code, but it ends with 400 error.
payload = {"password":"password","username":"username","options":{"warnBeforePasswordExpired": True,"multiOptionalFactorEnroll": True}}
with requests.Session() as s:
p = s.post('https://sso.johndeere.com/api/v1/authn', data=payload)
r = s.get("requested_url")
print(p)
I am unable get throw auth. Has anybody experience with breaking OKTA auth plugin using requests library?
Thanks
Best Regards
Merry Christmas and Welcome to Stackoverflow!
Firstly, an HTTP error code of 400 error means one or more settings is wrong at the client side. You can learn more about it here.
You seem to be missing out important headers configuration. You need to set the content-type header correctly otherwise the destination server won't be able to process your data.
Also, as a bonus point. You need to format your payload into a valid JSON string before sending out the request too.
import requests
import json
# Setup proper headers
headers = {
"accept": "application/json, text/plain, */*",
"content-type": "application/json; charset=UTF-8"
}
# Your body data here
payload = {"password":"password","username":"username","options":{"warnBeforePasswordExpired": True,"multiOptionalFactorEnroll": True}}
payload_json = json.dumps(payload) # Format it into a valid JSON str
with requests.Session() as s:
p = s.post('https://sso.johndeere.com/api/v1/authn', headers=headers, data=payload_json)
r = s.get("requested_url")
print(p.content)
I am writing a small application that interprets the http response of a request. I am writing the application in python. I have not found anything that allows me to send the body + headers stored in one file. I can send certain parts like the headers but not the entire request.
For example, if the request is:
GET /index.html HTTP/1.1
Host: localhost
Cookie: bob=lemon
I want to send this entire request in one go. How would I do this in python?
Check out the python requests library. https://requests.readthedocs.io/en/master/user/quickstart/#make-a-request
For the request above it would look something like
import requests
url = 'http://localhost:[YOUR PORT HERE]/'
cookies = {bob : lemon}
r = requests.get(url, cookies=cookies)
To check if you had a successful request you should get a 200 code from.
r.status_code
Check out the library for more, it is very extensive.
When downloading a pdf file from it's url I am observing a headers['content-type'] as 'text/html; charset=utf-8' when I need 'application/pdf'. Why is this doing it even when I am setting the Headers content-type?
Code example:
import requests
from requests.auth import HTTPBasicAuth
from pprint import pprint
file = 'url.pdf'
username = 'myusername'
password = 'mypassword'
headers = {'content-type': 'application/pdf', 'User-Agent': 'myUser-Agent'}
pdf_fname = 'new.pdf'
resp = requests.get(
file, headers=headers,
auth=HTTPBasicAuth(username, password),
proxies=proxyDict
)
with open(pdf_fname,'wb') as f:
f.write(resp.content)
pprint(resp.headers['content-type'])
GET requests do not have a content body, so have no need for a Content-Type header. Setting the header there is meaningless. HTTP servers generally will ignore the header on any GET requests they receive.
The header you observe is set by the HTTP server you contacted, and if the data you receive from the server is a PDF file, so a response with an incorrect Content-Type header, then that's entirely on the server, not on your code or on requests. Just ignore the header, or contact the administrators of the site you are contacting to ask them to correct the error.
However, if the server is actually sending you HTML, then you may want to save that HTML somewhere and open it in a browser to see what the server is trying to tell you. It may be a specific error message or login page. We can't tell you if this is the case here or not, we simply don't know how this specific website is designed to operate.
Also see another answer of mine that covers troubleshooting requests HTTP requests which differ from how a web browser is being treated for the same URLs.
I am a data analyst moonlighting in some web development, so apologies if this an obvious question, however I couldn't find a direct answer amongst previous questions or requests documentation.
I am trying to access data from an API that requires both basic auth and a content-type: application/json header. I have got this working in postman, but can't transfer to the python requests library - which returns 401.
I have tried passing in the login details (Username, password) as a tuple / list to the relevant dictionary key, however that doesn't work.
What am I missing?
import requests
headers={'Authorization': 'Basic <credentials>', 'Content-Type': 'application/json'}
r = requests.get("https://data.donorfy.com/api/v1/LFJZWVODI2/constituents/ExternalKey/1", headers=headers)
r.json()
I am trying to make an HTTP GET API call to one of my server, which support HTTP basic authentication using an API key in base64 encoding. so basically I want to add my authorization header in base64 encoding to my request.
The one method of authorization I know is:
>>> import requests
>>> r = requests.get('https://test.com/test-API-Gateway/v0/deployments', auth=('user', 'password'), verify=False)).text
>>> print r
{"statusCode":401,"statusMsg":Unauthorized,"result":[]}
But my server does not return anything, since it does not take id and password for authentication, rather it needs the base64 encoding header. Can you please tell me how to achieve this?
Thanks in advance.
The Python Requests library does allow you to add custom headers. You should be able to create the appropriate header (with your base64 encoding) and pass it as a parameter, like so:
import requests
url = 'https://test.com/test-API-Gateway/v0/deployments'
myheaders = {'my-header-param': 'somedata'}
r = requests.get(url, headers=myheaders, verify=False)).text
The related documentation can be found here.