I would like to modify following code to run use 'requests' module.
I have the following code which is working on a website:
def post(url, message, key, sign):
curl = pycurl.Curl()
curl.setopt(pycurl.URL, url)
curl.setopt(pycurl.SSL_VERIFYPEER, 0)
curl.setopt(pycurl.SSL_VERIFYHOST, 0)
buf = cStringIO.StringIO()
curl.setopt(pycurl.WRITEFUNCTION, buf.write)
curl.setopt(pycurl.POSTFIELDS, message)
curl.setopt(pycurl.HTTPHEADER, ['Key:' + key,
'Sign:' + (sign)])
curl.perform()
response = buf.getvalue()
buf.close()
return response
I tried accessing the website with requests and got rejected on invalid request values using following code:
def post(url, message, key, sign):
import requests
session = requests.session()
session.headers = {'Key': key, 'Sign': sign}
response = session.post(url, message)
return response
What am I doing wrong that these methods don't behave the same?
Thank you.
Using Pycurl:
POST /post HTTP/1.1
User-Agent: PycURL/7.32.0
Host: 127.0.0.1:4000
Accept: */*
Key:key
Sign:sign
Content-Length: 3
Content-Type: application/x-www-form-urlencoded
foo
With requests:
POST /post HTTP/1.1
Host: 127.0.0.1:4000
Accept-Encoding: identity
Content-Length: 3
Key: key
Sign: sign
foo
There are several differences which could lead to your error:
Missing User-Agent and Accept headers. This is because you overwrite the session.headers attribute which contains those default headers. Try this instead:
session.headers.update({'Key': key, 'Sign': sign})
Missing Content-Type header. I think you passed a string as the message parameter.
Requests doesn't know that this is application/x-www-form-urlencoded and therefore doesn't set the relevant header.
Either:
Set the header yourself
Better: pass requests a dictionary of your parameters. They will be encoded and declared correctly
Related
I want to use Python to simulate a login action which acquires some message sending via HTTP GET method. So I write something like this
from urllib.request import urlopen, Request
urlopen(Request(URL, data=data_for_verify.encode(), method='GET'))
The problem is, it doesn't do the same as a real login action which like this (scratch from Wireshark, HTTP printable data only)
GET /rjsdcctrl?mac%3dfcaa14ec56f3%26ipv4%3d1681312010%26ipv61%3d0%26ipv62%3d0%26ipv63%3d0%26ipv64%3d0%26product%3d33554432%26mainver%3d67108864%26subver%3d1610612736 HTTP/1.1
Accept: text/*
User-Agent: HttpCall
Accept-Language: en-us
Host: 10.0.6.251
Cache-Control: no-cache
And what my program did is:
GET / HTTP/1.1
Accept-Encoding: identity
Content-Type: application/x-www-form-urlencoded
Host: 10.0.6.251:80
User-Agent: Python-urllib/3.4
Connection: close
Content-Length: 161
rjsdcctrl?mac%3dfcaa14ec56f3%26ipv4%3d1681312010%26ipv61%3d0%26ipv62%3d0%26ipv63%3d0%26ipv64%3d0%26product%3d33554432%26mainver%3d67108864%26subver%3d1610612736
A real login action have the header comes first, and do not have the line GET / HTTP /1.1
or it just a header without content, and the first line GET contain the real request message. How can I simulate that using Python's urllib?
I use Python 3.4
You shouldn't use the data parameter if you don't want to send data as part of the body. Append the value to the URL:
full_url = "%s?%s" % (URL, data_for_verify.encode())
urlopen(full_url)
To extend #Daniel's answer, you can make use of urllib.urlencode method to prepare your get parameter string and also headers keyword argument to override default headers. So for example:
import urllib
url = 'http://www.example.com/'
data = {
'key1': 'value1',
'key2': 'value2',
'key3': 'value3'
}
headers = {
'Overriden-Header': 'Overriden Header Value'
}
## Update the url and make the actual requests
url = '%s?%s' % (url, urllib.urlencode(data))
response = urllib.urlopen(url, headers=headers)
How can I send HTTP requests as string with python? something like this:
r = """GET /hello.htm HTTP/1.1
User-Agent: Mozilla/4.0 (compatible; MSIE5.01; Windows NT)
Host: www.stackoverflow.com
Accept-Language: en-us
Accept-Encoding: gzip, deflate
Connection: Keep-Alive"""
answer = send(r)
print answer # gives me the response as string
Assuming python 3, it is recommended that you use urllib.request.
But since you specifically ask for providing the HTTP message as a string,
I assume you want to do low level stuff, so you can also use http.client:
import http.client
connection = http.client.HTTPConnection('www.python.org')
connection.request('GET', '/')
response = connection.getresponse()
print(response.status)
print(response.msg)
answer = response.read()
print(answer)
I'm trying to mimic a curl request I'm doing using python the call is. Note that
HTTPS Request
Ignoring SSL ceritification verification
My server is django
The curl command I used is
curl -k --dump-header - -H "Content-Type: application/json" -X POST --data '{"environment_name": "foo"}' https://localhost/api/v1/environment/
and the response from the server is successful
HTTP/1.1 201 CREATED
Date: Tue, 17 Jun 2014 00:59:59 GMT
Server: Server
Vary: Accept-Language,Cookie,User-Agent
Content-Language: en-us
Location: https://localhost/api/v1/environment/None/
Status: 201 CREATED
Content-Length: 0
Cneonction: close
Content-Type: text/html; charset=utf-8
However when I try to do a post request in python with 'requests' my script is
import json
data = {'enviornment_name' : 'foo'}
headers = {'Content-type' : 'application/json'}
response = requests.post("https://localhost/api/v1/environment", headers=headers, data=data, verify=False)
When running the script I get back a huge stack trace but the part in red is
E DecodeError: ('Received response with content-encoding: gzip, but failed to decode it.', error('Error -3 while decompressing: incorrect data check',))
I'm not sure why I can't my script to work via python
Your server is claiming to return gzip'd content which it is not. The response is returning Content-Encoding: gzip in the headers. This could be because in requests, by default, it sends Accept-Encoding: gzip, compress. You should try setting "Accept-Encoding" to None in your headers dictionary in addition to #Fabricator's suggestion in the comments.
Your headers dictionary will look like:
headers = {'Content-Type': 'application/json', 'Accept-Encoding': None}
And your call to requests will look like
requests.post(url, headers=headers, data=json.dumps(data), verify=False)
#Fabricator I need the verify=False however I noticed the one thing in my code that was an issue for the server I was using I needed the trailing '/' at the end of the URI. In addition I also needed the json.dumps(data) not json.dump(data) in case others are looking. Thanks for the help
I am using cakephp 2.4.5. I want to send a HTTP POST with URL parameters. I am using python 2.7 request module to send the HTTP POST. Please assume the payload is structured correctly as I have tested that part.
URL_post = http://127.0.0.1/webroot/TestFunc?identity_number=S111A/post
r = requests.post(URL_post, payload)
On the cakephp side, the controller looks something like this;
public function TestFunc($id=null)
{
$identity_number = $this->request->query['identity_number'];
$this->request->data['Model']['associated_id']=$identity_number;
$this->Model->saveAll($this->request->data, array('deep' => true));
}
I have tested that the query is not received correctly. However, if I am not using HTTP POST and just throwing in a normal URL, the query can be received correctly.
What have I done wrong?
The query part of the url is sent correctly:
import requests
requests.post('http://localhost/webroot/TestFunc?identity_number=S111A/post',
{'Model': 'data'})
The Request
POST /webroot/TestFunc?identity_number=S111A/post HTTP/1.1
Host: localhost
User-Agent: python-requests/2.2.1 CPython/3.4 Linux/3.2
Accept: */*
Accept-Encoding: gzip, deflate, compress
Content-Type: application/x-www-form-urlencoded
Content-Length: 10
Model=data
You could also make the requests using params:
requests.post('http://localhost/webroot/TestFunc',
data={'Model': 'data'},
params={'identity_number': 'S111A/post'})
The only difference is that S111A/post is sent as S111A%2Fpost (the same url in the end).
Look at http://docs.python-requests.org/en/latest/user/quickstart/#passing-parameters-in-urls.
payload = {"identity_number": "S111A/post"}
URL_post = "http://127.0.0.1/webroot/TestFunc"
req = requests.post(URL_post, params=payload)
print(req.status_code)
so I have made a request to a server with pythons request library. The code looks like this (it uses an adapter so it needs to match a certain pattern)
def getRequest(self, url, header):
"""
implementation of a get request
"""
conn = requests.get(url, headers=header)
newBody = conn.content
newHeader = conn.headers
newHeader['status'] = conn.status_code
response = {"Headers" : newHeader, "Body" : newBody.decode('utf-8')}
self._huddleErrors.handleResponseError(response)
return response
the header parameter I am parsing in is this
{'Authorization': 'OAuth2 handsOffMyToken', 'Accept': 'application/vnd.huddle.data+json'}
however I am getting an xml response back from the server. After checking fiddler I see the request being sent is:
Accept-Encoding: identity
Accept: */*
Host: api.huddle.dev
Authorization: OAuth2 HandsOffMyToken
Accept: application/vnd.huddle.data+json
Accept-Encoding: gzip, deflate, compress
User-Agent: python-requests/1.2.3 CPython/3.3.2 Windows/2008ServerR2
As we can see there are 2 Accept Headers! The requests library is adding in this Accept:* / * header which is throwing off the server. Does anyone know how I can stop this?
As stated in comments it seems this is a problem with the requests library in 3.3. In requests there are default headers (which can be found in the utils folder). When you don't specify your own headers these default headers are used. However if you specify your own headers instead requests tries to merge the headers together to make sure you have all the headers you need.
The problem shows its self in def request() method in sessions.py. Instead of merging all the headers it puts in its headers and then chucks in yours. For now I have just done the dirty hack of removing the accept header from the default headers found in util