Develop Reference

Python requests : trying to understand form data

i am new to requests in python and i'm trying to understand what's the data I send in the request and what i'm getting back.
Firstly, to understand better, i used the network inspector on chrome and uploaded a file on the website i'm going to send requests to later (the ultimate goal is to upload my file with requests).
It starts by opening a modal window with parameters so i'm guessing in python in something as easy as this (in python):
url = 'myurl'
params = {'whatever params i need'}
export = s.get(url, params=params)
if i print the status_code of this i get 200 so i'm guessing until then it's fine.
then it sends a post to the url without any parameters but with data like this (in python):
url = 'myurl'
data= {'confused'}
export = s.get(url, data=data)
here is where i'm getting a little confused. in the network inspector the data sent looks like this :
------WebKitFormBoundaryf2WTKCh05lDGbAAG
Content-Disposition: form-data; name="form[_token]"
Kmzz8c_N9qfuo8AZ1Pd1OFgaYzE9AFtitmaLkg0-y_g
------WebKitFormBoundaryf2WTKCh05lDGbAAG
Content-Disposition: form-data; name="form[importModule]"; filename="myfile.xml"
Content-Type: text/xml
------WebKitFormBoundaryf2WTKCh05lDGbAAG--
what does all this mean ? how am i supposed to write this in python ? And im guessing this "Kmzz8c_N9qfuo8AZ1Pd1OFgaYzE9AFtitmaLkg0-y_g" is the token, but how do i get in the first place too ?
thank you for your help and time !

You seem to be confused about "parameters" (query string parameters, "GET parameters", in any case the thing you use params= for in Requests) and form data.
What you see in the network inspector in the POST request is the form data (in particular, multipart/form-data data). If you inspect the form in the modal window, you'll probably find a hidden field with name="form[_token]", and a file field with name="form[importModule]".
To emulate that POST (with a file upload) with Requests, you'd do something like
s.post(
url="...",
data={
"form[_token]": "....",
},
files={
"form[importModule]": open("some_file.xlsx", "rb"),
},
)
To actually get the value for _token, you'd probably need to parse the response from the first GET request you do.

How to retrieve form data from Playwright Request object?

I am adding a custom route handler to a Playwright page and I am trying to inspect the request passed into the handler. For context here is a following code snippet:
def handler(route: Route, request: Request):
# Do things with `request`
...
await page.route('**/*', handler=handler)
For POST/PUT requests with a Content-Type of application/json, I have been able to successfully inspect the payload by using request.post_data_buffer. However, when the Content-Type is multipart/form-data, I have not been able locate where I can get the form data. All of the post_data, post_data_buffer, and post_data_json properties have a value of None, and I couldn't see anything else in the documentation which could contain the form_data.

The issue had nothing to do with really any details in my original post. The issue was I was using Chromium, and it is a known bug that post_data does not contain file/blob data.

Requests works and URLFetch doesn't

I'm trying to make a request to the particle servers in python in a google app engine app.
In my terminal, I can complete the request simply and successfully with requests as:
res = requests.get('https://api.particle.io/v1/devices', params={"access_token": {ACCESS_TOKEN}})
But in my app, the same thing doesn't work with urlfetch, which keeps telling me it can't find the access token:
url = 'https://api.particle.io/v1/devices'
payload = {"access_token": {ACCESS_TOKEN}}
form_data = urllib.urlencode(payload)
res = urlfetch.fetch(
url=url,
payload=form_data,
method=urlfetch.GET,
headers={
'Content-Type':
'application/x-www-form-urlencoded'
},
follow_redirects=False
)
I have no idea what the problem is, and no way to debug. Thanks!

In a nutshell, your problem is that in your urlfetch sample you're embedding your access token into the request body, and since you're issuing a GET request -which cannot carry any request body with them- this information gets discarded.
Why does your first snippet work?
Because requests.get() takes that optional params argument that means: "take this dictionary I give you, convert all its key/value pairs into a query string and append it to the main URL"
So, behind the curtains, requests.get() is building a string like this:
https://api.particle.io/v1/devices?access_token=ACCESS_TOKEN
That's the correct endpoint you should point your GET requests to.
Why doesn't your second snippet work?
This time, urlfetch.fetch() uses a different syntax than requests.get() (but equivalent nonetheless). The important bit to note here is that payload argument doesn't mean the same as our params argument that you used before in requests.get().
urlfetch.fetch() expects our query string -if any- to be already urlencoded into the URL (that's why urllib.urlencode() comes into play here). On the other hand, payload is where you should put your request body in case you were issuing a POST, PUT or PATCH request, but particle.io's endpoint is not expecting your OAuth access token to be there.
Something like this should work (disclaimer: not tested):
auth = {"access_token": {ACCESS_TOKEN}}
url_params = urllib.urlencode(auth)
url = 'https://api.particle.io/v1/devices?%s' % url_params
res = urlfetch.fetch(
url=url,
method=urlfetch.GET,
follow_redirects=False
)
Notice how now we don't need your previous Content-type header anymore, since we aren't carrying any content after all. Hence, headers parameter can be removed from this example call.
For further reference, take a look at urlfetch.fetch() reference and this SO thread that will hopefully give you a better insight into HTTP methods, parameters and request bodies than my poor explanation here.
PS: If particle.io servers support it (they should), you should move away from this authentication schema and carry your tokens in a Authorization: Bearer <access_token> header instead. Carrying access tokens in URLs is not a good idea because they are much more visible that way and tend to stay logged in servers, hence posing a security risk. On the other hand, in a TLS session all request headers are always encrypted so your auth tokens are well hidden there.

Ok, so, as it turns out, one cannot include a payload for a GET request using Urlfetch. Instead, one has to include the parameters in the url using the '?' syntax as follows:
url = 'https://api.particle.io/v1/devices'
url = url + '?access_token=' + {ACCESS_TOKEN}
res = urlfetch.fetch(
url=url,
method=urlfetch.GET,
follow_redirects=False
)
this worked for me.

Yggdrasil authentication with Python

I decided to try to make an automated login script for Minecraft. However, the new authentication API is stumping me. I can't find any mentions of the new functionality of the API on here. This is my code as it stands:
import requests
import json
data = json.dumps({"agent":{"name":"Minecraft","version":1},"username":"abcdef","password":"abcdef","clientToken":""})
headers = {'Content-Type': 'application/json'}
r = requests.post('https://authserver.mojang.com', data=data, headers=headers)
print (r.text)
Unfortunately, this returns:
{"error":"Method Not Allowed","errorMessage":"The method specified in the request is not allowed for the resource identified by the request URI"}
According to this resource on request format, this error means that I didn't correctly send a post request. However, I clearly declared requests.post(), so my first question is how am I incorrect, and what is the correct way to go about this?
My second question is, since I'm relatively new to Python and JSON, how would I replace the username and password fields with my own data, inside a variable?

You haven't specified an endpoint in your POST request, for example:
https://authserver.mojang.com/authenticate
The root of the website probably does not accept POST requests
http://wiki.vg/Authentication#Authenticate

Django is_ajax history back

I wrote a Django view that responses ether a text/html or a application/json depending on request.is_ajax().
So far so good, but when I use my browsers history buttons, I end up getting a JSON response rather than the HTML.
I can't figure out the problem. It's true an jQuery ajax request is getting the same url after the page was loaded, but that shouldn't end up in the history, or should it?
Thanks, Joe

If you send different content depending on request.is_ajax(), you need to send Vary: X-Requested-With to the browser. That way, the browser will be able to distinguish the two kinds of response based on the value of the X-Requested-With header on the request. You can do that via:
from django.views.decorators.vary import vary_on_headers
#vary_on_headers('X-Requested-With')
def yourview(request, ...):
pass