This question already has answers here:
Are HTTP headers case-sensitive?
(7 answers)
Closed 2 years ago.
I want to send the same header parameter twice with python requests. is it possible?
When I try to do the following request, the Requests lib from python ignores one of the passed headers even sending it with different cases:
Example:
import requests
url = "http://www.example.com"
headers = {"test":"test1",
"Test":"test2"}
req = requests.get(url, headers=headers)
print(req.request.headers)
HTTP header names are not case-sensitive, so requests handles this correctly. That the keys in the dictionary are case-insensitive is also sometimes mentioned in requests' docs, like Session.headers, Response.headers
requests stores the request headers in a dict, which means every header can only appear once (+ not case-sensitive). So without making changes to the requests library itself it won't be possible to send multiple headers with the same name.
Related
I've seen a couple of answers on this subject, but I still don't quite understand. Why is the RequestsCookieJar[] object sometimes empty? In the example below, for instance, it returns empty.
import requests
r = requests.get('https://brave.com/')
print(r.cookies)
Output:
<RequestsCookieJar[]>
The site (brave.com) does not use the header Set-Cookie. therefore, you have no cookies in your jar
I am doing a straight forward request as follows.
import requests
def user_transactions():
url = 'https://webapi.coinfloor.co.uk/v2/bist/XBT/GBP/user_transactions/'
data = {'key':'value'}
r = requests.post(url, data=data, auth=("some_username", "some_password") )
print(r.status_code)
print(r.text)
return
Even though data= is optional in the documents.
https://www.w3schools.com/python/ref_requests_post.asp
If i comment out the data variable then the routine returns a
status_code=415 error.
If i include in the data variable then the routine returns a status_code=200 success.
I have tried to look this up, for example here:
Python request gives 415 error while post data , but with no answer.
The question is: Why is it the case that [1] fails but [2] works ?
Yes, data is optional on the python side. The requests library will happily send a empty request to the server, as you can see. If the argument was not optional, the program would crash before sending a request so there would be no status code.
However, the server needs to be able to process the request. If it does not like what you sent for whatever reason, it might send back a 4xx status code, or otherwise not do what you expect.
In this case, it throws an error that the data is in invalid format. How can a empty request be in invalid format? Because the format is specified in a header. If you supply a data argumet requests will send data in urlencoded format, and specify in the header what format the data is in. If the data is empty, the request will be empty but the header will still be there. This site apparently requires the header to specify a data format it knows.
You can solve this in two ways, giving an empty object:
r = requests.post(url, data={}, auth=("some_username", "some_password") )
Or by explicitly specifying the header:
r = requests.post(url, auth=(...), headers={'Content-Type': 'application/x-www-form-urlencoded'})
Side note: You should not be using W3Schools as a source. It is frequently inaccurate and often recommends bad practices.
I think you are mistaking the documentation of the requests.post function signature with API documentation. It is saying that data is a keyword argument, not that the API optionally takes data.
It depends on the API endpoint you are trying to use. That endpoint must require data to be sent with the request. If you look at the documentation for the API you are using, it will mention what needs to be sent for a valid request.
This question already has answers here:
HTTP requests and JSON parsing in Python [duplicate]
(8 answers)
Closed 3 years ago.
I have a REST Api that only renders this data
data: "xxxxxxxxxxxxxxx"
I need to store this XXXXXXXXXX value in some variable using python, but I could only code and reach to the part where I get this value data:"xxxxxxxxxxxxxx"
my code is as follows
r = requests.get('url', headers=headers, verify=False)
logger.info(r)
which gives the output data: "xxxxxxxxxxxxxxx"
How can i fetch only XXXXXXXXXXXX from this json output ?
As simple as this could work for you.
r = requests.get('url', headers=headers, verify=False).json()
print(r['data'])
I have this long list of URL that I need to check response code of, where the links are repeated 2-3 times. I have written this script to check the response code of each URL.
connection =urllib.request.urlopen(url)
return connection.getcode()
The URL comes in XML in this format
< entry key="something" > url</entry>
< entry key="somethingelse" > url</entry>
and I have to associate the response code with the attribute Key so I don't want to use a SET.
Now I definitely don't want to make more than 1 request for the same URL so I was searching whether urlopen uses cache or not but didn't find a conclusive answer. If not what other technique can be used for this purpose.
You can store the urls in a dictionary (urls = {}) as you make a request and check if you have already made a req to that url later:
if key not in urls:
connection = urllib.request.urlopen(url)
urls[key] = url
return connection.getcode()
BTW if you make requests to the same urls repeatedly (multiple runs of the script), and need a persistent cache, i recommend using requests with requests-cache
Why don't you create a python set() of the URLs? That way each url is included only once.
How are you associating the URL with the key? A dictionary?
You can use a dictionary to map the URL to it's response and any other information you need to keep track of. If the URL is already in the dictionary then you know the response. So you have one dictionary:
url_cache = {
"url1" : ("response", [key1,key2])
}
If you need to organize things differently it shouldn't be too hard with another dictionary.
I have asked this question here about a Python command that fetches a URL of a web page and stores it in a variable. The first thing that I wanted to know then was whether or not the variable in this code contains the HTML code of a web-page:
from google.appengine.api import urlfetch
url = "http://www.google.com/"
result = urlfetch.fetch(url)
if result.status_code == 200:
doSomethingWithResult(result.content)
The answer that I received was "yes", i.e. the variable "result" in the code did contain the HTML code of a web page, and the programmer who was answering said that I needed to "check the Content-Type header and verify that it's either text/html or application/xhtml+xml". I've looked through several Python tutorials, but couldn't find anything about headers. So my question is where is this Content-Type header located and how can I check it? Could I send the content of that variable directly to my mailbox?
Here is where I got this code. It's on Google App Engines.
If you look at the Google App Engine documentation for the response object, the result of urlfetch.fetch() contains the member headers which contains the HTTP response headers, as a mapping of names to values. So, all you probably need to do is:
if result['Content-Type'] in ('text/html', 'application/xhtml+xml'):
# assuming you want to do something with the content
doSomethingWithXHTML(result.content)
else:
# use content for something else
doTheOtherThing(result.content)
As far as emailing the variable's contents, I suggest the Python email module.
for info on sending Content-Type header, see here: http://code.google.com/appengine/docs/python/urlfetch/overview.html#Request_Headers