Develop Reference

Send data through POST request

I had been using sockets, with Python, for some time ago and I'm trying to understand why this POST which should send some data on fields data1 and data2 do not work.
POST /method.php HTTP/1.1\r\nHost: localhost\r\nContent-Type: multipart/form-data\r\n\r\ndata1=something&data2= otherthing\r\n\r\n
What is the problem with this request?

There are several things wrong with your request:
POST /method.php HTTP/1.1
Host: localhost
Content-Type: multipart/form-data
data1=something&data2= otherthing
First, whenever a body is used within a HTTP request the length of the body must be known. This is typically done by given the length up-front with Content-length in the HTTP header although also chunked encoding might be used if the full length is not known up front. Your request does not do any of these which means the request is an invalid HTTP request.
Additionally you claim a Content-Type of multipart/form-data although your body is not of this type. With multipart/form-data your body would consist of several MIME parts separated by a text boundary and this boundary would need to have been declared in your Content-type header. The correct type for the body you show would be instead application/x-www-form-urlencoded.
Even with application/x-www-form-urlencoded the body is partly wrong. This type of body should be only pairs of key=value concatenated by &, i.e. there should be neither as space after a key as you have after data2= nor there should be new lines added after the end of the data as you have.
When removing all these problems you should probably send the following request:
body = "data1=something&data2=otherthing"
request = ("POST /method.php HTTP/1.1\r\n" + \
"Host: localhost\r\n" + \
"Content-Type: application/x-www-form-urlencoded\r\n" + \
"Content-Length: %d\r\n" + \
"\r\n%s") % (len(body),body)
But once you have send this request the trouble continues since getting the response correctly is complex too. Generally I recommend to not code your own HTTP handling unless you really know what you do but instead use existing libraries. While HTTP might look simple when just looking at a few example requests it is way more complex than it initially looks. And while your code might seem to work against specific servers it might fail with other servers.

It might be easier to use the requests library so your code would look something like this:
import requests
# Data
data = {
'data1':'something',
'data2':'otherthing'
}
# Custom headers
headers = {
'content-type': 'multipart/form-data'
}
# Get response from server
response = requests.post('http://localhost/', data=data, headers=headers)
# If you care about the response
print(response.json())
You can also send files and a whole lot of other stuff

Have you tried using the Requests library instead, example of a post request below
import requests
header = {"Content-Type": "multipart/form-data"}
data1="something"
data2= "otherthing"
session_requests = requests.session()
result = session_requests.post("http://localhost/", data=dict(data1, data2), headers=header)

Is there a way to get the full http request in pycurl?

c = pycurl.Curl()
c.setopt(c.URL, 'https://httpbin.org/post')
post_data = {'field': 'value'}
# Form data must be provided already urlencoded.
postfields = urlencode(post_data)
# Sets request method to POST,
# Content-Type header to application/x-www-form-urlencoded
# and data to send in request body.
c.setopt(c.POSTFIELDS, postfields)
c.perform()
c.close()
This is example from official doc.
I want to know what would be the http request looks like.
Is there a way to do that?

You could use the VERBOSE option, works like curl -v, will print request/response headers to stderr:
c.setopt(pycurl.VERBOSE, 1)
see more detail on pycurl doc.

slack api calls through python request library

I was making slack api calls through python library slackclient which is a wrapper around slack api. However, for some cases I need to make conventional api calls also with url and get/post method. I was trying to open a direct message channel with another user by my bot. The documentation - https://api.slack.com/methods/im.open says to "Present these parameters as part of an application/x-www-form-urlencoded querystring or POST body. application/json is not currently accepted."
Now in python, I can write,
url = 'https://slack.com/api/im.open'
headers = {'content-type':'x-www-form-urlencoded'}
data = {'token':BOT_TOKEN, 'user':user_id, 'include_locale':'true','return_im':'true'}
r= requests.post(url,headers,data )
print r.text
The message I get is {"ok":false,"error":"not_authed"}
I know the message is "not authed" although I use my bot token and another user id, my hunch is that I'm sending the request in wrong format because I just wrote it some way reading the documentation. I'm not sure how to exactly send these requests.
Any help?

since the Content-Type header is x-www-form-urlencoded sending data in form of dictionary does not work. you can try something like this.
import requests
url = 'https://slack.com/api/im.open'
headers = {'content-type': 'x-www-form-urlencoded'}
data = [
('token', BOT_TOKEN),
('user', user_id),
('include_locale', 'true'),
('return_im', 'true')
]
r = requests.post(url, data, **headers)
print r.text

The second parameter in requests.post is used for data, so in your request you're actually posting the headers dictionary. If you want to use headers you can pass arguments by name.
r= requests.post(url, data, headers=headers)
However this is not necessary in this case because 'x-www-form-urlencoded' is the default when posting form data.

How to perform HTTPS Calls in Pycurl?

I want to login into a website sending POST request using `pycurl' on a HTTPS page. I did the following:
import pycurl, urllib
curl = pycurl.Curl()
curl.setopt(pycurl.COOKIEFILE, "")
post = "_username=something&_password=somethingelse&_submit=Login"
curl.setopt(pycurl.URL, "https://www.example.com/login_check")
curl.setopt(pycurl.POST, 1)
curl.setopt(pycurl.POSTFIELDS, post)
curl.perform()
But I wasn't able to login. May be that was because my code made http call rather. Although I am able to perform the login sending 'POST' request with the same parameters in `POSTMAN'
So seems like login is right but I am doing something wrong implementing the same in python. Pl guide.

According to the pycURL documentation, your post data needs to be key value pairs and url encoded:
post_data = {'field': 'value'}
# Form data must be provided already urlencoded.
postfields = urlencode(post_data)
# Sets request method to POST,
# Content-Type header to application/x-www-form-urlencoded
# and data to send in request body.
c.setopt(c.POSTFIELDS, postfields)

multipart data POST using python requests: no multipart boundary was found

I have a form-data as well as file to be sent in the same POST. For ex, {duration: 2000, file: test.wav}. I saw the many threads here on multipart/form-data posting using python requests. They were useful, especially this one.
My sample request is as below:
files = {'file': ('wavfile', open(filename, 'rb'))}
data = {'duration': duration}
headers = {'content-type': 'multipart/form-data'}
r = self.session.post(url, files=files, data=data, headers=headers)
But when I execute the above code, I get this error:
5:59:55.338 Dbg 09900 [DEBUG] Resolving exception from handler [null]: org.springframework.web.multipart.MultipartException: Could not parse multipart servlet request; nested exception is org.apache.commons.fileupload.FileUploadException: the request was rejected because no multipart boundary was found.
So my questions are: 1) How can I see the content of the request being sent? Couldn't use wireshark, its not across the network.
2) why is the boundary missing in the encoded data? Did I miss anything, please point out.

You should NEVER set that header yourself. We set the header properly with the boundary. If you set that header, we won't and your server won't know what boundary to expect (since it is added to the header). Remove your custom Content-Type header and you'll be fine.

Taking out the Content-Type header with explicit "multipart/form-data" worked!

To specifically add boundary add following in header :
headers = {
'content-type': 'multipart/form-data; boundary=ebf9f03029db4c2799ae16b5428b06bd'
}