I'm having some difficulties parsing multipart form data when using the django-rest-framework. I've set up a minimal view to just echo back the request data:
from rest_framework.views import APIView
from rest_framework.response import Response
from rest_framework.parsers import MultiPartParser, FormParser
class FileUpload(APIView):
parser_classes = (MultiPartParser, FormParser, )
def post(self, request, format=None, *args, **kwargs):
return Response({'raw': request.data, 'data': request._request.POST,
'files': str(request._request.FILES)})
I am expecting that raw (slightly badly named I admit) contains effectively the same data as request._request.POST and request._request.FILES.
If I POST to the view with Content-Type= application/x-www-form-urlencoded this works as expected:
$ http -v --form POST http://localhost:8000/upload/api/ course=3 name=name
POST /upload/api/ HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Content-Length: 18
Content-Type: application/x-www-form-urlencoded; charset=utf-8
Host: localhost:8000
User-Agent: HTTPie/0.9.2
course=3&name=name
HTTP/1.0 200 OK
Allow: POST, OPTIONS
Content-Type: application/json
Date: Thu, 17 Dec 2015 16:52:37 GMT
Server: WSGIServer/0.2 CPython/3.4.3
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN
{
"data": {
"course": "3",
"name": "name"
},
"files": "<MultiValueDict: {}>",
"raw": {
"course": "3",
"name": "name"
}
}
However if I post with Content-Type=multipart/form-data I get the following:
$ http -v --form POST http://localhost:8000/upload/api/ file#~/Projects/lms/manage.py course=3 name=name
POST /upload/api/ HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Content-Length: 577
Content-Type: multipart/form-data; boundary=634ec7c7e89a487b89c1c07c0d24744c
Host: localhost:8000
User-Agent: HTTPie/0.9.2
--634ec7c7e89a487b89c1c07c0d24744c
Content-Disposition: form-data; name="course"
3
--634ec7c7e89a487b89c1c07c0d24744c
Content-Disposition: form-data; name="name"
name
--634ec7c7e89a487b89c1c07c0d24744c
Content-Disposition: form-data; name="file"; filename="manage.py"
#!/usr/bin/env python
import os
import sys
if __name__ == "__main__":
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "config.settings.local")
from django.core.management import execute_from_command_line
execute_from_command_line(sys.argv)
--634ec7c7e89a487b89c1c07c0d24744c--
HTTP/1.0 200 OK
Allow: POST, OPTIONS
Content-Type: application/json
Date: Thu, 17 Dec 2015 16:55:44 GMT
Server: WSGIServer/0.2 CPython/3.4.3
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN
{
"data": {
"course": "3",
"name": "name"
},
"files": "<MultiValueDict: {'file': [<InMemoryUploadedFile: manage.py ()>]}>",
"raw": {}
}
Am I missing something here? I am using HTTPIE to generate the requests here but the same behaviour exists with curl so I'm pretty sure that is not the problem. I am using djangorestframework==3.3.0 and Django==1.8.4
EDIT:
It seems that PUTing to the url (with an otherwise identical request) achieves the desired result:
$ http -v --form PUT http://localhost:8000/upload/api/ file#~/Projects/lms/manage.py course=3 name=name
PUT /upload/api/ HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Content-Length: 577
Content-Type: multipart/form-data; boundary=98feb59a8abe4bfb95a7321f536ed800
Host: localhost:8000
User-Agent: HTTPie/0.9.2
--98feb59a8abe4bfb95a7321f536ed800
Content-Disposition: form-data; name="course"
3
--98feb59a8abe4bfb95a7321f536ed800
Content-Disposition: form-data; name="name"
name
--98feb59a8abe4bfb95a7321f536ed800
Content-Disposition: form-data; name="file"; filename="manage.py"
#!/usr/bin/env python
import os
import sys
if __name__ == "__main__":
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "config.settings.local")
from django.core.management import execute_from_command_line
execute_from_command_line(sys.argv)
--98feb59a8abe4bfb95a7321f536ed800--
HTTP/1.0 200 OK
Allow: POST, PUT, OPTIONS
Content-Type: application/json
Date: Thu, 17 Dec 2015 18:10:34 GMT
Server: WSGIServer/0.2 CPython/3.4.3
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN
{
"data": {},
"files": "<MultiValueDict: {}>",
"raw": "<QueryDict: {'name': ['name'], 'course': ['3'], 'file': [<InMemoryUploadedFile: manage.py ()>]}>"
}
So I could just use PUT. That is however not ideal as the client does not control what the file is named or where it is located on the server. POST is more appropriate in that sense. In any sense, I don't see why PUT works when POST doesn't.
It is a known issue for the version you're using. Upgrading django rest framework to the latest version will solve the problem. However, you can PUT the request as a workaround.
Related
I am trying to send a POST request with Python to upload a file. I'm converting the following sample code to Python but I'm not familiar with how to set this up.
POST /path/to/upload/script HTTP/1.0
Connection: Keep-Alive
User-Agent: My Client App v1.0
Host:
https://bulksell.ebay.com/ws/eBayISAPI.dll?FileExchangeUpload
Content-type: multipart/form-data;
boundary=THIS_STRING_SEPARATES
Content-Length: 256
--THIS_STRING_SEPARATES
Content-Disposition: form-data; name="token"
12345678987654321
--THIS_STRING_SEPARATES
Content-Disposition: form-data; name="file";
filename="listings.csv"
Content-Type: text/csv
... contents of listings.csv ...
--THIS_STRING_SEPARATES
I think I need to set the headers as follows:
headers = {
"Connection": "Keep-Alive",
"User-Agent": "My Client App v1.0",
"Host": "https://bulksell.ebay.com/ws/eBayISAPI.dll?FileExchangeUpload"
"Content-type": "multipart/form-data;"
"Content-Length": "256",
"Host": "https://bulksell.ebay.com/ws/eBayISAPI.dll?FileExchangeUpload",
...
}
Do I need to include these --THIS_STRING_SEPERATES strings?
How do I include my token here? The example just sends it alone.
What would be the correct way to format this for a request.post?
Thank you.
I have made a post requests with multipart/form-data; boundary=a1c2469c-2a1f-48e6-8f1d-311f8650c855 And I get this
POST /api/feed/upload-resource HTTP/1.1
App-Id: 15762288
Version-Name: 3.26.0.451
User-Agent: Right-Android/3.26.0.451
Content-Type: multipart/form-data; boundary=a1c2469c-2a1f-48e6-8f1d-311f8650c855
Content-Length: 75412
Connection: Keep-Alive
Accept-Encoding: gzip
--a1c2469c-2a1f-48e6-8f1d-311f8650c855
Content-Disposition: form-data; name="h"
Content-Length: 4
1080
--a1c2469c-2a1f-48e6-8f1d-311f8650c855
Content-Disposition: form-data; name="w"
Content-Length: 4
1080
--a1c2469c-2a1f-48e6-8f1d-311f8650c855
Content-Disposition: form-data; name="image"; filename="/storage/emulated/0/Android/data/com.xiaoyu.rightone/tiny/tiny-738-2018-08-03-15-32-53.jpg"
Content-Type: text/plain
Content-Length: 74910
If I want to simulate this request using Python requests package How can I do that.
I have checked the document, and I have tried use file parameter like this
with open(path, 'rb') as f:
files = {
"h": "1495",
"w": "840",
"filename": ("image", f.read()),
}
r = requests.post(url, files=files)
However in my situation, I always get an error like upload_resource_empty.
This should work:
import requests
files = {
'image': ('file_name.jpg', open('file.jpg', 'rb'), 'text/plain'),
'w': (None, '123'),
'h': (None, '222')
}
response = requests.post('url', files=files)
The tuples in the dictionary have the following format: ('filename', fileobj, 'content_type', custom_headers)
I'm trying to write a python script that would help me install a theme remotely. Unfortunately, the upload part doesn't play nice, trying to do it with requests' POST helpers.
The HTTP headers of a successful upload look like this:
http://127.0.0.1/wordpress/wp-admin/update.php?action=upload-theme
POST /wordpress/wp-admin/update.php?action=upload-theme HTTP/1.1
Host: 127.0.0.1
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:53.0) Gecko/20100101 Firefox/53.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Content-Type: multipart/form-data; boundary=---------------------------2455316848522
Content-Length: 2580849
Referer: http://127.0.0.1/wordpress/wp-admin/theme-install.php
Cookie: wordpress_5bd7a9c61cda6e66fc921a05bc80ee93=admin%7C1497659497%7C4a1VklpOs93uqpjylWqckQs80PccH1QMbZqn15lovQu%7Cee7366eea9b5bc9a9d492a664a04cb0916b97b0d211e892875cec86cf43e2f9d; wordpress_test_cookie=WP+Cookie+check; wordpress_logged_in_5bd7a9c61cda6e66fc921a05bc80ee93=admin%7C1497659497%7C4a1VklpOs93uqpjylWqckQs80PccH1QMbZqn15lovQu%7C9949f19ef5d900daf1b859c0bb4e2129cf86d6a970718a1b63e3b9e56dc5e710; wp-settings-1=libraryContent%3Dbrowse; wp-settings-time-1=1497486698
Connection: keep-alive
Upgrade-Insecure-Requests: 1
-----------------------------2455316848522: undefined
Content-Disposition: form-data; name="_wpnonce"
b1467671e0
-----------------------------2455316848522
Content-Disposition: form-data; name="_wp_http_referer"
/wordpress/wp-admin/theme-install.php
-----------------------------2455316848522
Content-Disposition: form-data; name="themezip"; filename="oedipus_theme.zip"
Content-Type: application/octet-stream
PK
HTTP/1.1 200 OK
Date: Thu, 15 Jun 2017 01:33:25 GMT
Server: Apache/2.4.25 (Win32) OpenSSL/1.0.2j PHP/7.1.1
X-Powered-By: PHP/7.1.1
Expires: Wed, 11 Jan 1984 05:00:00 GMT
Cache-Control: no-cache, must-revalidate, max-age=0
X-Frame-Options: SAMEORIGIN
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: text/html; charset=UTF-8
----------------------------------------------------------
To create a simple session for WP, in order to use later for uploads:
global wp_session
def wpCreateSession(uname, upassword, site_link):
"""
:param uname: Username for the login.
:param upaswword: Password for the login.
:param site_link: Site to login on.
:return: Returns a sessions for the said website.
"""
global wp_session
wp_session = requests.session()
wp_session.post(site_link, data={'log' : uname, 'pwd' : upassword})
To upload the said file to WP, using the wp_session global:
def wpUploadTheme(file_name):
global wp_session
try:
with open(file_name, 'rb') as up_file:
r = wp_session.post('http://127.0.0.1/wordpress/wp-admin/update.php', files = {file_name: up_file})
print "Got after try."
finally:
up_file.close()
And this last bit is where it doesn't work, the upload is not successful and I get returned to WordPress' basic 404.
I have also tried requests_toolbelt MultiPart_Encoder to no avail.
Question: 'requests' POST file fails when trying to upload
Check your files dict, your dict is invalid
files = {file_name: up_file}
Maybe you need a full blown files dict, for instance:
files = {'themezip': ('oedipus_theme.zip',
open('oedipus_theme.zip', 'rb'),
'application/octet-stream', {'Expires': '0'})}
From docs.python-requests.org
files = {'file': open('test.jpg', 'rb')}
requests.post(url, files=files)
From SO Answer Upload Image using POST form data in Python-requests
I have some simple code, like this:
import json
from bottle import route, request,run
#route('/process_json',methods='POST')
def data_process():
data = json.loads(request.data)
username = data['username']
password = data['password']
run(host='localhost', port=8080, debug=True)
I would like to send a data in json format, like this:
$ curl -i -H "Accept: application/json" -H "Content-Type: application/json" -X POST -d
'{"username":"fooba","password":"1234"}' url
However, I have this problem:
File "/usr/lib/python2.7/dist-packages/bottle.py", line 862,
in _handle return route.call(**args)
File "/usr/lib/python2.7/dist-packages/bottle.py", line 1729,
in wrapper rv = callback(*a, **ka)
File "testtest.py", line 5,
in data_process data = json.loads(request.data)
File "/usr/lib/python2.7/dist-packages/bottle.py", line 1391,
in getattr raise AttributeError('Attribute %r not defined.' % name)
AttributeError: Attribute 'data' not defined
I also tried add at the beginning like here(http://bottlepy.org/docs/dev/tutorial.html#html-form-handling):
return '''
Username:
Password:
'''
But it also doesn't work.
It is good, you have provided the bottle application code. It was broken.
Following modification works:
import json
from bottle import route, request, run
#route('/process_json', method="POST")
def data_process():
data = json.load(request.body)
print "data", data
username = data['username']
password = data['password']
run(host='localhost', port=8080, debug=True)
Testing with curl
$ curl -i -H "Accept: application/json" -H "Content-Type: application/json" -X POST -d '{"username": "fooba", "password": "1234"}' http://localhost:8080/process_json
HTTP/1.0 200 OK
Date: Mon, 14 Jul 2014 16:18:25 GMT
Server: WSGIServer/0.1 Python/2.7.6
Content-Length: 0
Content-Type: text/html; charset=UTF-8
Testing with HTTPie
$ http POST http://localhost:8080/process_json username=jan password=pswd
HTTP/1.0 200 OK
Content-Length: 0
Content-Type: text/html; charset=UTF-8
Date: Mon, 14 Jul 2014 16:15:16 GMT
Server: WSGIServer/0.1 Python/2.7.6
Actually, I was playing with your example to find out, how it would look like in http command. It is really much simpler and using --verbose we may check, it really forms valid JSON payload:
$ http --verbose POST http://localhost:8080/process_json Accept:application/json
Content_type:application/json username=jan password=pswd
POST /process_json HTTP/1.1
Accept: application/json
Accept-Encoding: gzip, deflate
Authorization: Basic amFuOnZsY2luc2t5
Content-Length: 39
Content-Type: application/json; charset=utf-8
Content_type: application/json
Host: localhost:8080
User-Agent: HTTPie/0.8.0
{
"password": "pswd",
"username": "jan"
}
HTTP/1.0 200 OK
Content-Length: 0
Content-Type: text/html; charset=UTF-8
Date: Mon, 14 Jul 2014 16:21:02 GMT
Server: WSGIServer/0.1 Python/2.7.6
The shortest form is:
$ http :8080/process_json username=jan password=pswd
HTTP/1.0 200 OK
Content-Length: 0
Content-Type: text/html; charset=UTF-8
Date: Mon, 14 Jul 2014 16:25:12 GMT
Server: WSGIServer/0.1 Python/2.7.6
as http://localhost:8008 can be shortened to :8080 (the same applies to curl)
and if there is a payload, default method is POST.
I've followed the following article in an attempt to setup Apache2 caching in order to use it with Django on Ubuntu 12.10 with mod_wsgi. I want Apache to cache some requests for me.
http://www.howtoforge.com/caching-with-apaches-mod_cache-on-ubuntu-10.04
From the article I enabled the modules and setup the following php script to test the caching. The caching works just fine - I only get a new timestamp after 5 minutes.
vi /var/www/cachetest.php
<?php
header("Cache-Control: must-revalidate, max-age=300");
header("Vary: Accept-Encoding");
echo time()."<br>";
?>
Now in my django response, I return an HttpResponse object after setting the appropriate headers the same way:
# Create a Response Object with the content to return and set it's
response = HttpResponse("%s"%(output_display))
response['Cache-Control'] = 'must-revalidate, max-age=20'
response['Vary'] = 'Accept-Encoding'
return response
The caching with the Django request doesn't work at all. I've used Firefox's LiveHeaders to examine the HTTP response headers.
For the example link above and the PHP script the headers look like:
http://localhost/cachetest.php
GET /cachetest.php HTTP/1.1
Host: localhost
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:19.0) Gecko/20100101 Firefox/19.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
Cache-Control: max-age=0
HTTP/1.1 200 OK
Date: Sun, 10 Mar 2013 02:29:32 GMT
Server: Apache/2.2.22 (Ubuntu)
X-Powered-By: PHP/5.4.6-1ubuntu1.1
Cache-Control: must-revalidate, max-age=300
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 34
Connection: close
Content-Type: text/html
----------------------------------------------------------
For my Django Request - the caching doesn't work, it always forces the lengthy operation to complete the response - just like re-loading the php request above with F5. Using the FireFox plugin I seem to be writing the correct headers:
http://localhost/testdjango/testdjango/
GET /testdjango/testdjango/ HTTP/1.1
Host: localhost
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:19.0) Gecko/20100101 Firefox/19.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
HTTP/1.1 200 OK
Date: Sun, 10 Mar 2013 02:32:41 GMT
Server: Apache/2.2.22 (Ubuntu)
Vary: Accept-Encoding
Cache-Control: must-revalidate, max-age=20
Content-Encoding: gzip
Connection: close
Transfer-Encoding: chunked
Content-Type: text/html; charset=utf-8
----------------------------------------------------------
What am I doing wrong? How can I get the django caching to work like the php script? Thanks!
This seems to be your problem:
Transfer-Encoding: chunked
It means a 'streaming response', in terms of mod_mem_cache. And, according to the docs:
By default, a streamed response will not be cached unless it has a
Content-Length header.
You can solve it by setting the MCacheMaxStreamingBuffer directive.