Develop Reference

Python requests : trying to understand form data

i am new to requests in python and i'm trying to understand what's the data I send in the request and what i'm getting back.
Firstly, to understand better, i used the network inspector on chrome and uploaded a file on the website i'm going to send requests to later (the ultimate goal is to upload my file with requests).
It starts by opening a modal window with parameters so i'm guessing in python in something as easy as this (in python):
url = 'myurl'
params = {'whatever params i need'}
export = s.get(url, params=params)
if i print the status_code of this i get 200 so i'm guessing until then it's fine.
then it sends a post to the url without any parameters but with data like this (in python):
url = 'myurl'
data= {'confused'}
export = s.get(url, data=data)
here is where i'm getting a little confused. in the network inspector the data sent looks like this :
------WebKitFormBoundaryf2WTKCh05lDGbAAG
Content-Disposition: form-data; name="form[_token]"
Kmzz8c_N9qfuo8AZ1Pd1OFgaYzE9AFtitmaLkg0-y_g
------WebKitFormBoundaryf2WTKCh05lDGbAAG
Content-Disposition: form-data; name="form[importModule]"; filename="myfile.xml"
Content-Type: text/xml
------WebKitFormBoundaryf2WTKCh05lDGbAAG--
what does all this mean ? how am i supposed to write this in python ? And im guessing this "Kmzz8c_N9qfuo8AZ1Pd1OFgaYzE9AFtitmaLkg0-y_g" is the token, but how do i get in the first place too ?
thank you for your help and time !

You seem to be confused about "parameters" (query string parameters, "GET parameters", in any case the thing you use params= for in Requests) and form data.
What you see in the network inspector in the POST request is the form data (in particular, multipart/form-data data). If you inspect the form in the modal window, you'll probably find a hidden field with name="form[_token]", and a file field with name="form[importModule]".
To emulate that POST (with a file upload) with Requests, you'd do something like
s.post(
url="...",
data={
"form[_token]": "....",
},
files={
"form[importModule]": open("some_file.xlsx", "rb"),
},
)
To actually get the value for _token, you'd probably need to parse the response from the first GET request you do.

How to send entire http request from file?

I am writing a small application that interprets the http response of a request. I am writing the application in python. I have not found anything that allows me to send the body + headers stored in one file. I can send certain parts like the headers but not the entire request.
For example, if the request is:
GET /index.html HTTP/1.1
Host: localhost
Cookie: bob=lemon
I want to send this entire request in one go. How would I do this in python?

Check out the python requests library. https://requests.readthedocs.io/en/master/user/quickstart/#make-a-request
For the request above it would look something like
import requests
url = 'http://localhost:[YOUR PORT HERE]/'
cookies = {bob : lemon}
r = requests.get(url, cookies=cookies)
To check if you had a successful request you should get a 200 code from.
r.status_code
Check out the library for more, it is very extensive.

Send data through POST request

I had been using sockets, with Python, for some time ago and I'm trying to understand why this POST which should send some data on fields data1 and data2 do not work.
POST /method.php HTTP/1.1\r\nHost: localhost\r\nContent-Type: multipart/form-data\r\n\r\ndata1=something&data2= otherthing\r\n\r\n
What is the problem with this request?

There are several things wrong with your request:
POST /method.php HTTP/1.1
Host: localhost
Content-Type: multipart/form-data
data1=something&data2= otherthing
First, whenever a body is used within a HTTP request the length of the body must be known. This is typically done by given the length up-front with Content-length in the HTTP header although also chunked encoding might be used if the full length is not known up front. Your request does not do any of these which means the request is an invalid HTTP request.
Additionally you claim a Content-Type of multipart/form-data although your body is not of this type. With multipart/form-data your body would consist of several MIME parts separated by a text boundary and this boundary would need to have been declared in your Content-type header. The correct type for the body you show would be instead application/x-www-form-urlencoded.
Even with application/x-www-form-urlencoded the body is partly wrong. This type of body should be only pairs of key=value concatenated by &, i.e. there should be neither as space after a key as you have after data2= nor there should be new lines added after the end of the data as you have.
When removing all these problems you should probably send the following request:
body = "data1=something&data2=otherthing"
request = ("POST /method.php HTTP/1.1\r\n" + \
"Host: localhost\r\n" + \
"Content-Type: application/x-www-form-urlencoded\r\n" + \
"Content-Length: %d\r\n" + \
"\r\n%s") % (len(body),body)
But once you have send this request the trouble continues since getting the response correctly is complex too. Generally I recommend to not code your own HTTP handling unless you really know what you do but instead use existing libraries. While HTTP might look simple when just looking at a few example requests it is way more complex than it initially looks. And while your code might seem to work against specific servers it might fail with other servers.

It might be easier to use the requests library so your code would look something like this:
import requests
# Data
data = {
'data1':'something',
'data2':'otherthing'
}
# Custom headers
headers = {
'content-type': 'multipart/form-data'
}
# Get response from server
response = requests.post('http://localhost/', data=data, headers=headers)
# If you care about the response
print(response.json())
You can also send files and a whole lot of other stuff

Have you tried using the Requests library instead, example of a post request below
import requests
header = {"Content-Type": "multipart/form-data"}
data1="something"
data2= "otherthing"
session_requests = requests.session()
result = session_requests.post("http://localhost/", data=dict(data1, data2), headers=header)

How to perform HTTPS Calls in Pycurl?

I want to login into a website sending POST request using `pycurl' on a HTTPS page. I did the following:
import pycurl, urllib
curl = pycurl.Curl()
curl.setopt(pycurl.COOKIEFILE, "")
post = "_username=something&_password=somethingelse&_submit=Login"
curl.setopt(pycurl.URL, "https://www.example.com/login_check")
curl.setopt(pycurl.POST, 1)
curl.setopt(pycurl.POSTFIELDS, post)
curl.perform()
But I wasn't able to login. May be that was because my code made http call rather. Although I am able to perform the login sending 'POST' request with the same parameters in `POSTMAN'
So seems like login is right but I am doing something wrong implementing the same in python. Pl guide.

According to the pycURL documentation, your post data needs to be key value pairs and url encoded:
post_data = {'field': 'value'}
# Form data must be provided already urlencoded.
postfields = urlencode(post_data)
# Sets request method to POST,
# Content-Type header to application/x-www-form-urlencoded
# and data to send in request body.
c.setopt(c.POSTFIELDS, postfields)

How do you to post a diff to ReviewBoard through its API?

I have been struggling to post a diff to ReviewBoard through their API. I've managed to login to the server and create a new post, but I've failed to post correctly the contents of the diff file.
I'm new to writing this kind of application, but my goal is to have a one step script to:
diff a file (pre-commit) with the svn repository,
add a review request to ReviewBoard and post the diff from the current file,
May be later, the script can be part of a svn pre-commit hook.
My python attempt looks like:
import urllib.request
import urllib.parse
import os.path
... login to the reviewboard server with
urllib.request.HTTPBasicAuthHandler ...
diff_path = '/path/to/file'
diff_name = 'my.diff'
diff_path = os.path.join(diff_path, diff_name)
diff_val = open(diff_path,'r')
# load the diff into the http data POST request
diff_header = \
'-- SoMe BoUnDaRy \n' \
+ 'Content-Disposition: form-data; name=path; filename=' \
+ '"' + diff_name + '"\n\n' \
+ diff_val.read() + '\n' \
+ '-- SoMe BoUnDaRy --'
data ={'path': diff_header, 'basedir': '/path/to/file/in/rep'}
print( data['path'] )
data = urllib.parse.urlencode(data)
data = data.encode('utf-8')
opener.open( \
'http://xxx.xxx.x.xxx/api/review-requests/26/diffs/', data)
With this code I get a BAD REQUEST(400) error, specifically: "One or more fields had errors" (105).
I'm aware that there are some libraries out there that can talk with the ReviewBoard API. I'm also aware that post-review exists. I'd rather not have to distribute to the other developers another python library and post-review seems less flexible when diffing files from multiple locations.
From the suggestion below, I've add the server response here:
CREATING PASSWD MANAGER...
CREATING PASSWD MANAGER... done
CREATING PASSWD HANDLER...
CREATING PASSWD HANDLER... done
CREATING URL OPENER...
CREATING URL OPENER... done
LOADING DIFF...
send: b'POST /api/review-requests/26/diffs/ HTTP/1.1\r\nAccept-Encoding:
identity\r\nContent-Length: 723\r\nHost: xxx.xxx.x.xxx\r\nContent-Type:
application/x-www-form-urlencoded\r\nConnection: close\r\nUser-Agent:
[empty no username+password] Python-urllib/3.2\r\n\r\
npath=--+SoMe+BoUnDaRy+++%...[the rest of my post]
reply: 'HTTP/1.1 401 UNAUTHORIZED\r\n'
header: Date header: Server header: Content-Language header: Expires header:
Vary header: Cache-Control header: WWW-Authenticate header:
Content-Length header: Last-Modified header: Connection header:
Content-Type send: b'POST /api/review-requests/26/diffs/
HTTP/1.1\r\nAccept-Encoding: identity\r\nContent-Length: 723\r\nHost:
xxx.xxx.x.xxx\r\nUser-Agent: Python-urllib/3.2\r\nConnection:
close\r\nContent-Type: application/x-www-form-urlencoded\r\nAuthorization:
Basic [with username+password]\r\n\r\npath=
--+SoMe+BoUnDaRy+++%0AContent-Disposition%...
reply: 'HTTP/1.1 400 BAD REQUEST\r\n'
header: Date header: Server header: Content-Language header: Expires header:
Vary header: Cache-Control header: Set-Cookie header: Content-Length header:
Last-Modified header: Connection header: Content-Type HTTPError thrown
At first glance my guess is that something is happening to my password handler. I'm not sure what is happening to it. Just in case, this is how I'm generate my authentication:
manager_passwd = urllib.request.HTTPPasswordMgr()
manager_passwd.add_password(...)
handler_passwd = urllib.request.HTTPBasicAuthHandler(manager_passwd)
opener = urllib.request.build_opener(handler_passwd)
The authentication seems to working. I've tested it by create a new review post. So it is when I post the diff that the authentication fails.

Reviewboard have already a python tool for posting diff with their API, it's called postreview.py. You can found it at :
http://reviewboard.googlecode.com/svn/trunk/wxpostreview/postreview.py
Grab and use their ReviewBoardServer for login and post a diff !
(In addition, in your request, the authentification is required yes, but also the cookie file. That's why you need 2 requests (one for login and get the cookie, another one for sending the diff.))