Develop Reference

How to get the request header sent by requests.get()?

I'd like to know what a HTTP GET request header is sent by requests.get() from the client side.
requests.get('http://localhost:9000')
The request header sent by the above python command monitored by netcat is the following. However, I don't find a way to directly monitor the HTTP GET request header sent at the client side.
GET / HTTP/1.1
Host: localhost:9000
Connection: keep-alive
Accept-Encoding: gzip, deflate
Accept: */*
User-Agent: python-requests/2.18.4
import requests
import sys
req = requests.Request('GET', 'localhost:9000')
print req.headers
prepared = req.prepare()
s = requests.Session()
page = s.send(prepared)
The request header sent by the above python command monitored by netcat is the following.
GET / HTTP/1.1
Host: localhost:9000
Accept-Encoding: identity
Also, req.headers can be used to monitor the header. It is not exactly the same as the header sent as it does not contain the Accept-Encoding header. Also, the HTTP GET request header sent by this way is also different from that of the first way.
$ ./main.py
{}
Is there a method to directly monitor what HTTP GET header is sent in the first way?
Also, why the requests send by the two methods are not the same? Isn't it better to make them consistent to avoid possible confusion?

I'd like to know what a HTTP GET request header is sent by requests.get() from the client side
If I got it right, you want to view headers, that were actually sent by requests.get().
You can access them by using .request.headers attributes:
import requests
r = requests.get("http://example.com")
print(r.request.headers)

Python - sending http requests as strings

How can I send HTTP requests as string with python? something like this:
r = """GET /hello.htm HTTP/1.1
User-Agent: Mozilla/4.0 (compatible; MSIE5.01; Windows NT)
Host: www.stackoverflow.com
Accept-Language: en-us
Accept-Encoding: gzip, deflate
Connection: Keep-Alive"""
answer = send(r)
print answer # gives me the response as string

Assuming python 3, it is recommended that you use urllib.request.
But since you specifically ask for providing the HTTP message as a string,
I assume you want to do low level stuff, so you can also use http.client:
import http.client
connection = http.client.HTTPConnection('www.python.org')
connection.request('GET', '/')
response = connection.getresponse()
print(response.status)
print(response.msg)
answer = response.read()
print(answer)

How to send/receive URL parameters in a HTTP POST correctly?

I am using cakephp 2.4.5. I want to send a HTTP POST with URL parameters. I am using python 2.7 request module to send the HTTP POST. Please assume the payload is structured correctly as I have tested that part.
URL_post = http://127.0.0.1/webroot/TestFunc?identity_number=S111A/post
r = requests.post(URL_post, payload)
On the cakephp side, the controller looks something like this;
public function TestFunc($id=null)
{
$identity_number = $this->request->query['identity_number'];
$this->request->data['Model']['associated_id']=$identity_number;
$this->Model->saveAll($this->request->data, array('deep' => true));
}
I have tested that the query is not received correctly. However, if I am not using HTTP POST and just throwing in a normal URL, the query can be received correctly.
What have I done wrong?

The query part of the url is sent correctly:
import requests
requests.post('http://localhost/webroot/TestFunc?identity_number=S111A/post',
{'Model': 'data'})
The Request
POST /webroot/TestFunc?identity_number=S111A/post HTTP/1.1
Host: localhost
User-Agent: python-requests/2.2.1 CPython/3.4 Linux/3.2
Accept: */*
Accept-Encoding: gzip, deflate, compress
Content-Type: application/x-www-form-urlencoded
Content-Length: 10
Model=data
You could also make the requests using params:
requests.post('http://localhost/webroot/TestFunc',
data={'Model': 'data'},
params={'identity_number': 'S111A/post'})
The only difference is that S111A/post is sent as S111A%2Fpost (the same url in the end).

Look at http://docs.python-requests.org/en/latest/user/quickstart/#passing-parameters-in-urls.
payload = {"identity_number": "S111A/post"}
URL_post = "http://127.0.0.1/webroot/TestFunc"
req = requests.post(URL_post, params=payload)
print(req.status_code)

How to translate url requests using pycurl to python-requests?

I would like to modify following code to run use 'requests' module.
I have the following code which is working on a website:
def post(url, message, key, sign):
curl = pycurl.Curl()
curl.setopt(pycurl.URL, url)
curl.setopt(pycurl.SSL_VERIFYPEER, 0)
curl.setopt(pycurl.SSL_VERIFYHOST, 0)
buf = cStringIO.StringIO()
curl.setopt(pycurl.WRITEFUNCTION, buf.write)
curl.setopt(pycurl.POSTFIELDS, message)
curl.setopt(pycurl.HTTPHEADER, ['Key:' + key,
'Sign:' + (sign)])
curl.perform()
response = buf.getvalue()
buf.close()
return response
I tried accessing the website with requests and got rejected on invalid request values using following code:
def post(url, message, key, sign):
import requests
session = requests.session()
session.headers = {'Key': key, 'Sign': sign}
response = session.post(url, message)
return response
What am I doing wrong that these methods don't behave the same?
Thank you.

Using Pycurl:
POST /post HTTP/1.1
User-Agent: PycURL/7.32.0
Host: 127.0.0.1:4000
Accept: */*
Key:key
Sign:sign
Content-Length: 3
Content-Type: application/x-www-form-urlencoded
foo
With requests:
POST /post HTTP/1.1
Host: 127.0.0.1:4000
Accept-Encoding: identity
Content-Length: 3
Key: key
Sign: sign
foo
There are several differences which could lead to your error:
Missing User-Agent and Accept headers. This is because you overwrite the session.headers attribute which contains those default headers. Try this instead:
session.headers.update({'Key': key, 'Sign': sign})
Missing Content-Type header. I think you passed a string as the message parameter.
Requests doesn't know that this is application/x-www-form-urlencoded and therefore doesn't set the relevant header.
Either:
Set the header yourself
Better: pass requests a dictionary of your parameters. They will be encoded and declared correctly

requests python lib adding in Accept header

so I have made a request to a server with pythons request library. The code looks like this (it uses an adapter so it needs to match a certain pattern)
def getRequest(self, url, header):
"""
implementation of a get request
"""
conn = requests.get(url, headers=header)
newBody = conn.content
newHeader = conn.headers
newHeader['status'] = conn.status_code
response = {"Headers" : newHeader, "Body" : newBody.decode('utf-8')}
self._huddleErrors.handleResponseError(response)
return response
the header parameter I am parsing in is this
{'Authorization': 'OAuth2 handsOffMyToken', 'Accept': 'application/vnd.huddle.data+json'}
however I am getting an xml response back from the server. After checking fiddler I see the request being sent is:
Accept-Encoding: identity
Accept: */*
Host: api.huddle.dev
Authorization: OAuth2 HandsOffMyToken
Accept: application/vnd.huddle.data+json
Accept-Encoding: gzip, deflate, compress
User-Agent: python-requests/1.2.3 CPython/3.3.2 Windows/2008ServerR2
As we can see there are 2 Accept Headers! The requests library is adding in this Accept:* / * header which is throwing off the server. Does anyone know how I can stop this?

As stated in comments it seems this is a problem with the requests library in 3.3. In requests there are default headers (which can be found in the utils folder). When you don't specify your own headers these default headers are used. However if you specify your own headers instead requests tries to merge the headers together to make sure you have all the headers you need.
The problem shows its self in def request() method in sessions.py. Instead of merging all the headers it puts in its headers and then chucks in yours. For now I have just done the dirty hack of removing the accept header from the default headers found in util