I've got problem while trying to post the file to the server. I'm trying to make file upload script to server, this server is very 'Sensitive to correctness post request'
I debugged page that is sending the file to server and browser send this (TextView):
POST http://example.com/post HTTP/1.1
Host: example.com
Connection: keep-alive
Content-Length: 20625
Accept: application/json, text/javascript, */*; q=0.01
Origin: http://example.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.104 Safari/537.36
Content-Type: multipart/form-data; boundary=----WebKitFormBoundarykGHBkXoER9gNuVna
Referer: http://example.com/foo
Accept-Encoding: gzip, deflate
Accept-Language: pl-PL,pl;q=0.8,en-US;q=0.6,en;q=0.4,pt;q=0.2
------WebKitFormBoundarykGHBkXoER9gNuVna
Content-Disposition: form-data; name="files[]"; filename="file.zip"
Content-Type: application/octet-stream
...raw file data...
------WebKitFormBoundarykGHBkXoER9gNuVna--
However, my script is sending this (TextView):
POST http://example.com/post HTTP/1.1
Host: example.com
Connection: keep-alive
Accept-Encoding: gzip, deflate
Accept: */*
User-Agent: python-requests/2.18.1
Content-Length: 20604
--f8c266cf436941019c5a80c7d4779a57
Content-Disposition: form-data; name="files[]"; filename="file.zip"
Content-Type: application/zip
...raw file data...
--f8c266cf436941019c5a80c7d4779a57--
With causes error on server, additional note: this error started when I changed files=files to data=files
Current Code:
files = MultipartEncoder({'files[]': (filename, open(local_path,'rb'), mimetype)})
UploadFile = requests.post(self.UploadURL, data=files, allow_redirects=False)
Working code:
files = {'files[]': (filename, open(local_path,'rb'), mimetype)}
UploadFile = requests.post(self.UploadURL, files=files, allow_redirects=False)
I'm using MultipartEncoder to allow sending huge files.
I see that biggest mismatch is "boundary", but why this 'boundary' is generating in working code but in Current code not?
How to fix that?
You are not setting the Content-Type header, the MultipartEncoder provides it for you:
files = MultipartEncoder({'files[]': (filename, open(local_path,'rb'), mimetype)})
UploadFile = requests.post(
self.UploadURL, data=files, allow_redirects=False,
headers={'Content-Type': files.content_type})
The header must come from the multi-part encoding, because it is responsible for picking the boundary used to deliniate the various MIME parts in the multipart response. In your upload that's:
--f8c266cf436941019c5a80c7d4779a57
but it is generated at random each time your code runs. The header provided would look like:
Content-Type: multipart/form-data; boundary=--f8c266cf436941019c5a80c7d4779a57
Related
I am beginner in python. I would like to parse a website but the header shows the content type text/x-gwt-rpc; charset=utf-8 and the request payload...
7|0|4|https://kekeke.cc/com.liquable.hiroba.home.gwt.HomeModule/|53263EDF7F9313FDD5BD38B49D3A7A77|com.liquable.hiroba.gwt.client.square.IGwtSquareService|getNoOfCrowd|1|2|3|4|0|
Request:
POST /com.liquable.hiroba.gwt.server.GWTHandler/squareService HTTP/1.1
Host: kekeke.cc
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:59.0) Gecko/20100101 Firefox/59.0
Accept: */*
Accept-Language: zh-TW,zh;q=0.8,en-US;q=0.5,en;q=0.3
Accept-Encoding: gzip, deflate, br
Referer: https://kekeke.cc/
Content-Type: text/x-gwt-rpc; charset=utf-8
X-GWT-Permutation: 8F22796231EB8C8312C5D1BB10451262
X-GWT-Module-Base: https://kekeke.cc/com.liquable.hiroba.home.gwt.HomeModule/
Content-Length: 177
DNT: 1
Connection: keep-alive
Can anyone tell me how to make post request in python?
I found the solution. It can be solved by simply using r = requests.post(url, "7|0|4|https://kekeke.cc/com.liquable.hiroba.home.gwt.HomeModule/|53263EDF7F9313FDD5BD38B49D3A7A77|com.liquable.hiroba.gwt.client.square.IGwtSquareService|getNoOfCrowd|1|2|3|4|0|", headers=headers). In many online tutorials, they teach the post request by using data in form of {data:data} to submit post request only. However, it can be done by submitting data in form of string in some cases.
I want to make a program where I can send HTTP post requests and respond.
So, I want to send THIS post:
POST https: //example.com/index.php?s=&&app=box&module=ajax§ion=coreAjax&secure_key=&type=submit&lastid=87311&global=1 HTTP/1.1
Host: example.com
Connection: keep-alive
Content-Length: 10
Accept: text/javascript, text/html, application/xml, text/xml, */*
X-Prototype-Version: 1.7.2
Origin: https://example.com
X-Requested-With: XMLHttpRequest
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36
Content-type: application/x-www-form-urlencoded; charset=UTF-8
Referer: https://x.com/
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.8
Cookie: cookieconsent_status=dismiss;
And then enter the request body:
message= # Which I will make: "message= %s" % (messagex))
But I do not know how to send them and can´t seem to find any way online, could someone help please?
The main parts are:
import requests # you have to install this library, with pip for example
# define your custom headers (as many as you want)
headers = {
'X-Prototype-Version': '1.7.2'
}
# define your URL params (!= of the body of the POST request)
params = {
'your_first_param': 'its_value',
'your_second_param': 'its_value'
}
# define the body of the POST request
data = {
'message' : 'your message'
}
# send the POST request
response = requests.post('https://example.com/index.php', params=params, data=data, headers=headers)
# here is the response
print response.text
Hope that helps.
I am using Google App Engine with python 2.7. And there is need to generate in-memory xls-file and send it to user for downloading.
I found amount of topics in web, but any of them can't help me.
Related topics that I've tried to use: 1) this is with Blobs, I tried at first, 2) without Blob, 3) with force-download MIME type, also I've tried to use googlecloudstorage (can't find links to topics).
Here is my code:
import StringIO
class ExcelHandler(BaseHandler):
def post(self):
"""Save members to excel document and send to user"""
sheet = pyexcel.Sheet([[1, 2], [3, 4]])
filesheet = StringIO.StringIO()
sheet.save_to_memory('xls', filesheet)
filesheet.close()
self.response.write(sheet)
self.response.headers['Content-Type'] = 'application/force-download'
self.response.headers['Content-Transfer-Encoding'] = 'utf-8'
self.response.headers['Content-Disposition'] = 'attachment; filename=test.xlsx'
The problem is in sending response (not in creating file). I tried different 'Content-Type':
'application/vnd.ms-excel',
'application/download',
'application/force-download',
'application/octet-stream',
'application/vnd.openxmlformats - officedocument.spreadsheetml.sheet'
But the best response I've achieved is as on picture:
I can't enforce my browser to start downloading data from server. I guess there may be something in my Request that should say to server 'Hey, I want to download', but it is only my thoughts, I've not found anything about that. Will appreciate any help!
Here is also my Request:
POST /reg/excel HTTP/1.1
Host: 0.0.0.0:8080
Connection: keep-alive
Content-Length: 0
Accept: */*
Origin: http://0.0.0.0:8080
X-Requested-With: XMLHttpRequest
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.106 Safari/537.36
Referer: http://0.0.0.0:8080/competition?dbKey=agpkZXZ- dG1tb3NjchgLEgtDb21wZXRpdGlvbhiAgICAgICgCww
Accept-Encoding: gzip, deflate
Accept-Language: ru-RU,ru;q=0.8,en-US;q=0.6,en;q=0.4
and Response at debugger:
HTTP/1.1 200 OK
content-disposition: attachment; filename=test.xlsx
content-transfer-encoding: utf-8
cache-control: no-cache
content-type: application/force-download
Content-Length: 64
Server: Development/2.0
Date: Sun, 02 Oct 2016 15:36:20 GMT
EDIT 1: (try answer by voscausa)
Try this:
output = StringIO.StringIO()
.......
self.response.headers[b'Content-Type'] = b'application/vnd.ms-excel; charset=utf-8'
self.response.headers[b'Content-Disposition'] = b'attachment; filename=test.xlsx'
self.response.write(output.getvalue())
We have our own private cloud where I have installed OpenStack Swift. I have a working node (proxy and storage) that allows me to store and retrieve if I use the openstack and swift python cli to store and retrieve files. Additionally I am able to use the python API on a remote machine to store and retrieve files.
The root of my question is how to debug temp url and crossdomain filter issues. Is there a way to turn on detailed debug logging for these filters?
I have the default logging set to
log_name = swift
log_facility = LOG_LOCAL0
log_level = DEBUG
The situation I am trying to troubleshoot is as follows. When I try and use temp url and cross domain (for CORS), I get a 401. I debugged the code and it appears to be a invalid HMAC error. Based on research, this appears to be a date time issue where the client and the server have missed matched times. However both are running the ntpd service so the time should be in sync.
For CORS, it appears that preflight OPTIONS request is succeeding. The subsequent PUT is failing with a 401....
"No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://blahost' is therefore not allowed access. The response had HTTP status code 401."
The strange part is the OPTIONS request is returning "access-control-allow-origin" instead of 'Access-Control-Allow-Origin'... the case is off.
Preflight request:
OPTIONS /v1/AUTH_99cf99f26aaa4b2c923806231b03334c/436/88b6d895-6dbf-4f29-904d-96c9b7959016?temp_url_sig=4a953c34372e37b2a22bb31fb0581a7eb7f02cee&temp_url_expires=1441508891 HTTP/1.1
Host: 23.253.200.41:8080
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
Access-Control-Request-Method: PUT
Origin: http://blahhost
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36
Access-Control-Request-Headers: accept, content-type
Accept: */*
Referer: http://blahost/binder/436/site/419/folder/17560/file
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8
Preflight Response:
HTTP/1.1 200 OK
access-control-allow-origin: http://blahost
access-control-allow-methods: HEAD, GET, PUT, POST, COPY, OPTIONS, DELETE
access-control-allow-headers: content-type, accept
Allow: HEAD, GET, PUT, POST, COPY, OPTIONS, DELETE
Content-Length: 0
X-Trans-Id: tx9e359777dfb94148858cd-0055eba012
Date: Sun, 06 Sep 2015 02:08:18 GMT
Connection: keep-alive
Subsequent PUT request(notice it is missing the Access-Control-Allow-Origin)
PUT /v1/AUTH_99cf99f26aaa4b2c923806231b03334c/436/88b6d895-6dbf-4f29-904d-96c9b7959016?temp_url_sig=4a953c34372e37b2a22bb31fb0581a7eb7f02cee&temp_url_expires=1441508891 HTTP/1.1
Host: 23.253.200.41:8080
Connection: keep-alive
Content-Length: 16231
Pragma: no-cache
Cache-Control: no-cache
Accept: */*
Origin: http://blahost
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36
Content-Type: application/pdf
Referer: http://blahost/binder/436/site/419/folder/17560/file
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8
I would appreciate any advice on how to troubleshoot.
Thanks
Greg
I'm trying to make a POST request of multipart/form-data using mechanize, here's what it looks like from firefox live http header when I actually make a post:
http://example.com/new/example
POST /new/example HTTP/1.1
Host: example.com
User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:13.0) Gecko/20100101 Firefox/13.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
Referer: http://example.com/new/example
Cookie: tmgioct=c32MbAGn1sTuZrH8etPqVNU5; __qca=P0-495598852-1339139301054; __utma=189990958.911848588.1339139302.1339556345.1339561805.32; __utmz=189990958.1339139302.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); logged_in=1; tog_appearance_fieldset=fieldset_open; __utmc=189990958; pfu=42375294; pfp=h2YrFoaTr5LtrVys8PMmKNdyuoeA9FNLakxGzrJK; pfe=1371048319; __utmb=189990958.5.10.1339561805
Content-Type: multipart/form-data; boundary=---------------------------41184676334
Content-Length: 2947
-----------------------------41184676334
Content-Disposition: form-data; name="UPLOAD_IDENTIFIER"
0ad3af1c502c7cb59577b01720ee58ff014810c4
-----------------------------41184676334
Content-Disposition: form-data; name="post[state]"
2
-----------------------------41184676334
blahblahblahblah....
-----------------------------41184676334--
And here's my code:
browser = mechanize.Browser()
url = "http://example.com/new/example"
header = {
'User-Agent': 'Mozilla/5.0 (Windows NT 5.1; rv:13.0) Gecko/20100101 Firefox/13.0',
'Referer': 'http://example.com/new/example',
'Content-Type': 'multipart/form-data; boundary=---------------------------41184676334'
}
data = "-----------------------------41184676334\rContent-Disposition: form-data; name="UPLOAD_IDENTIFIER"\r\r0ad3af1c502c7cb59577b01720ee58ff014810c4\r-----------------------------41184676334\rContent-Disposition: form-data; name="post[state]"\r\r2\r-----------------------------41184676334\rblahblahblahblah....\r\r-----------------------------41184676334--\r"
req = urllib2.Request(url, data, header)
response = browser.open(req, timeout = 30)
response.close()
I don't know why it does NOT work. Anybody knows? Please help me out.
By the way, does it have something to do with boundary? I use random numbers in above code.
From the MIME media types RFC 2046:
The canonical form of any MIME "text" subtype MUST always represent a
line break as a CRLF sequence.
Your code uses carriage returns ('\r') only; you need to add line feeds (\n) as well.
browser.form.enctype = "application/x-www-form-urlencoded"
Ended up using requests module to do the task. It turned out to be more convenient and reliable.
You can check out this page for details: POST a Multipart-Encoded File