i am new to requests in python and i'm trying to understand what's the data I send in the request and what i'm getting back.
Firstly, to understand better, i used the network inspector on chrome and uploaded a file on the website i'm going to send requests to later (the ultimate goal is to upload my file with requests).
It starts by opening a modal window with parameters so i'm guessing in python in something as easy as this (in python):
url = 'myurl'
params = {'whatever params i need'}
export = s.get(url, params=params)
if i print the status_code of this i get 200 so i'm guessing until then it's fine.
then it sends a post to the url without any parameters but with data like this (in python):
url = 'myurl'
data= {'confused'}
export = s.get(url, data=data)
here is where i'm getting a little confused. in the network inspector the data sent looks like this :
------WebKitFormBoundaryf2WTKCh05lDGbAAG
Content-Disposition: form-data; name="form[_token]"
Kmzz8c_N9qfuo8AZ1Pd1OFgaYzE9AFtitmaLkg0-y_g
------WebKitFormBoundaryf2WTKCh05lDGbAAG
Content-Disposition: form-data; name="form[importModule]"; filename="myfile.xml"
Content-Type: text/xml
------WebKitFormBoundaryf2WTKCh05lDGbAAG--
what does all this mean ? how am i supposed to write this in python ? And im guessing this "Kmzz8c_N9qfuo8AZ1Pd1OFgaYzE9AFtitmaLkg0-y_g" is the token, but how do i get in the first place too ?
thank you for your help and time !
You seem to be confused about "parameters" (query string parameters, "GET parameters", in any case the thing you use params= for in Requests) and form data.
What you see in the network inspector in the POST request is the form data (in particular, multipart/form-data data). If you inspect the form in the modal window, you'll probably find a hidden field with name="form[_token]", and a file field with name="form[importModule]".
To emulate that POST (with a file upload) with Requests, you'd do something like
s.post(
url="...",
data={
"form[_token]": "....",
},
files={
"form[importModule]": open("some_file.xlsx", "rb"),
},
)
To actually get the value for _token, you'd probably need to parse the response from the first GET request you do.
Related
body_1 = {"landowner": "asdf#gmail.com","title": report_name,"content": "asdf","project": projectID, }
url_1 = 'http://' + IP + ':8005/projects/notifications/'
response_1 = requests.post(url_1, data=body_1, headers=headers)
For this, I am getting an error 500, when above I have used a similar process to post and it has worked.
I even wrote this code in javascript, and that works but the python equivalent does not.
I tried to add console logs, and everything works up until the response, where it says code 500.
I tried to follow along with javascript as well and try and model it after that too, but that did not work either.
You are getting the error because you are not sending the data as JSON encoded data. When sending an array as data with a request use the json keyword argument. This means your request will look like;
response_1 = requests.post(url_1, json=body_1, headers=headers)
In the example above data= was changed to json=.
By using the json argument the data is encoded to JSON, and the Content-Type header is set to application/json.
Learn more about the json= keyword argument in the requests documentation
If this doesn't fix your issue please add the error output message for the 500 error code.
I am trying to update an already saved form on a system using HTTP requests. Due to the server configuration for the third party app we use, updating by POST requires sending a fully filled out payload every single time.
I want to get round this by recovering the form data already present on the server and converting it into a dictionary. Then changing any values I need and reposting to make changes sever side.
The application we use sends a POST request when the save button is clicked for a particular form.
Here I send a post request with no payload.
[This simulates pressing the save button and is also the point where dev tools shows me a the payload I want to capture]
post_test = self.session.post(url_to_retrieve_from)
I thought that now I should be able to print the output, which should resemble what Google Dev tools Form data captures.
print(post_test.text)
This just gives me html found on the webpage.
If Dev Tools can get this from the server then I should also be able to?
Example of Data I am trying to get via requests:
Form Data
If Dev Tools can get this from the server then I should also be able to?
Yes, of course. In requests you pass form data in data keyword:
import requests
url = 'http://www.example.com'
data = {
'name': 'value',
}
response = requests.post(url, data=data)
You can get the data you sent with a request from the response in this way:
import requests
response = requests.post('http://your_url', data=data) # send request
body = response.request.body
parsed_data = dict(data.split('=') for data in body.split('&')) # parse request body
Here you can find more information about data argument
In the documentation, in the class requests.Response we can find the attribute:
request = None
The PreparedRequest object to which this is a response.
In requests.PreparedRequest class we can read:
body = None
request body to send to the server.
I have been browsing stackoverflow for the past couple of days and have been looking at a lot of different videos and forums, but I can't get this to work for some reason. I'm trying to automatically add an item to cart on https://www.toytokyo.com/medicom-toy-kaws-together-black/ and I even get the correct 200 response code, but when check the shopping cart it says that its empty.
Here is the Request Payload that it needs.
------WebKitFormBoundary2abcTSnRV9XhBx4h
Content-Disposition: form-data; name="action"
add
------WebKitFormBoundary2abcTSnRV9XhBx4h
Content-Disposition: form-data; name="product_id"
4806
------WebKitFormBoundary2abcTSnRV9XhBx4h
Content-Disposition: form-data; name="qty[]"
1
------WebKitFormBoundary2abcTSnRV9XhBx4h--
and here is what I'm doing to send the POST request.
payload = {'action': 'add', 'product_id': 4806, 'qty[]': 1}
get = requests.get("https://www.toytokyo.com/medicom-toy-kaws-together-black/")
post = requests.post("https://www.toytokyo.com/remote/v1/cart/add", data=payload)
print(post.status_code, post.content)
get = requests.get("https://www.toytokyo.com/cart.php")
print(get.status_code, get.text)
I'm not sure if I'm doing something wrong, but I get the correct response from what I can tell.
EDIT: ANSWER BELOW
Just for anyone that might stumble across this later on, I took the advice of the people who commented below and created a variable called session and assigned it using session = requests.Session() which allows your program to persist across every new request that you send. the session variable also has all of the same methods as the request itself. So I just replaced everything that used requests and replaced it with session.
You perform the correct POST/GET call, however you need to take into account the fact that you also need some way to track your "session". Likely on a real page, cookies are used to track the contents of your cart. As a result, when you request your cart contents, you will need to include this cookie. To do so, add cookies to your code using a requests session:
s = requests.Session() # cookies are stored in the session
payload = {'action': 'add', 'product_id': 4806, 'qty[]': 1}
get = s.get("https://www.toytokyo.com/medicom-toy-kaws-together-black/")
post = s.post("https://www.toytokyo.com/remote/v1/cart/add", data=payload)
print(post.status_code, post.content)
get = s.get("https://www.toytokyo.com/cart.php")
print(get.status_code, get.text)
I had been using sockets, with Python, for some time ago and I'm trying to understand why this POST which should send some data on fields data1 and data2 do not work.
POST /method.php HTTP/1.1\r\nHost: localhost\r\nContent-Type: multipart/form-data\r\n\r\ndata1=something&data2= otherthing\r\n\r\n
What is the problem with this request?
There are several things wrong with your request:
POST /method.php HTTP/1.1
Host: localhost
Content-Type: multipart/form-data
data1=something&data2= otherthing
First, whenever a body is used within a HTTP request the length of the body must be known. This is typically done by given the length up-front with Content-length in the HTTP header although also chunked encoding might be used if the full length is not known up front. Your request does not do any of these which means the request is an invalid HTTP request.
Additionally you claim a Content-Type of multipart/form-data although your body is not of this type. With multipart/form-data your body would consist of several MIME parts separated by a text boundary and this boundary would need to have been declared in your Content-type header. The correct type for the body you show would be instead application/x-www-form-urlencoded.
Even with application/x-www-form-urlencoded the body is partly wrong. This type of body should be only pairs of key=value concatenated by &, i.e. there should be neither as space after a key as you have after data2= nor there should be new lines added after the end of the data as you have.
When removing all these problems you should probably send the following request:
body = "data1=something&data2=otherthing"
request = ("POST /method.php HTTP/1.1\r\n" + \
"Host: localhost\r\n" + \
"Content-Type: application/x-www-form-urlencoded\r\n" + \
"Content-Length: %d\r\n" + \
"\r\n%s") % (len(body),body)
But once you have send this request the trouble continues since getting the response correctly is complex too. Generally I recommend to not code your own HTTP handling unless you really know what you do but instead use existing libraries. While HTTP might look simple when just looking at a few example requests it is way more complex than it initially looks. And while your code might seem to work against specific servers it might fail with other servers.
It might be easier to use the requests library so your code would look something like this:
import requests
# Data
data = {
'data1':'something',
'data2':'otherthing'
}
# Custom headers
headers = {
'content-type': 'multipart/form-data'
}
# Get response from server
response = requests.post('http://localhost/', data=data, headers=headers)
# If you care about the response
print(response.json())
You can also send files and a whole lot of other stuff
Have you tried using the Requests library instead, example of a post request below
import requests
header = {"Content-Type": "multipart/form-data"}
data1="something"
data2= "otherthing"
session_requests = requests.session()
result = session_requests.post("http://localhost/", data=dict(data1, data2), headers=header)
I'm trying to create a super-simplistic Virtual In / Out Board using wx/Python. I've got the following code in place for one of my requests to the server where I'll be storing the data:
data = urllib.urlencode({'q': 'Status'})
u = urllib2.urlopen('http://myserver/inout-tracker', data)
for line in u.readlines():
print line
Nothing special going on there. The problem I'm having is that, based on how I read the docs, this should perform a Post Request because I've provided the data parameter and that's not happening. I have this code in the index for that url:
if (!isset($_POST['q'])) { die ('No action specified'); }
echo $_POST['q'];
And every time I run my Python App I get the 'No action specified' text printed to my console. I'm going to try to implement it using the Request Objects as I've seen a few demos that include those, but I'm wondering if anyone can help me explain why I don't get a Post Request with this code. Thanks!
-- EDITED --
This code does work and Posts to my web page properly:
data = urllib.urlencode({'q': 'Status'})
h = httplib.HTTPConnection('myserver:8080')
headers = {"Content-type": "application/x-www-form-urlencoded",
"Accept": "text/plain"}
h.request('POST', '/inout-tracker/index.php', data, headers)
r = h.getresponse()
print r.read()
I am still unsure why the urllib2 library doesn't Post when I provide the data parameter - to me the docs indicate that it should.
u = urllib2.urlopen('http://myserver/inout-tracker', data)
h.request('POST', '/inout-tracker/index.php', data, headers)
Using the path /inout-tracker without a trailing / doesn't fetch index.php. Instead the server will issue a 302 redirect to the version with the trailing /.
Doing a 302 will typically cause clients to convert a POST to a GET request.