How to send a multipart/form-data with requests in python? How to send a file, I understand, but how to send the form data by this method can not understand.
Basically, if you specify a files parameter (a dictionary), then requests will send a multipart/form-data POST instead of a application/x-www-form-urlencoded POST. You are not limited to using actual files in that dictionary, however:
>>> import requests
>>> response = requests.post('http://httpbin.org/post', files=dict(foo='bar'))
>>> response.status_code
200
and httpbin.org lets you know what headers you posted with; in response.json() we have:
>>> from pprint import pprint
>>> pprint(response.json()['headers'])
{'Accept': '*/*',
'Accept-Encoding': 'gzip, deflate',
'Connection': 'close',
'Content-Length': '141',
'Content-Type': 'multipart/form-data; '
'boundary=c7cbfdd911b4e720f1dd8f479c50bc7f',
'Host': 'httpbin.org',
'User-Agent': 'python-requests/2.21.0'}
Better still, you can further control the filename, content type and additional headers for each part by using a tuple instead of a single string or bytes object. The tuple is expected to contain between 2 and 4 elements; the filename, the content, optionally a content type, and an optional dictionary of further headers.
I'd use the tuple form with None as the filename, so that the filename="..." parameter is dropped from the request for those parts:
>>> files = {'foo': 'bar'}
>>> print(requests.Request('POST', 'http://httpbin.org/post', files=files).prepare().body.decode('utf8'))
--bb3f05a247b43eede27a124ef8b968c5
Content-Disposition: form-data; name="foo"; filename="foo"
bar
--bb3f05a247b43eede27a124ef8b968c5--
>>> files = {'foo': (None, 'bar')}
>>> print(requests.Request('POST', 'http://httpbin.org/post', files=files).prepare().body.decode('utf8'))
--d5ca8c90a869c5ae31f70fa3ddb23c76
Content-Disposition: form-data; name="foo"
bar
--d5ca8c90a869c5ae31f70fa3ddb23c76--
files can also be a list of two-value tuples, if you need ordering and/or multiple fields with the same name:
requests.post(
'http://requestb.in/xucj9exu',
files=(
('foo', (None, 'bar')),
('foo', (None, 'baz')),
('spam', (None, 'eggs')),
)
)
If you specify both files and data, then it depends on the value of data what will be used to create the POST body. If data is a string, only it willl be used; otherwise both data and files are used, with the elements in data listed first.
There is also the excellent requests-toolbelt project, which includes advanced Multipart support. It takes field definitions in the same format as the files parameter, but unlike requests, it defaults to not setting a filename parameter. In addition, it can stream the request from open file objects, where requests will first construct the request body in memory:
from requests_toolbelt.multipart.encoder import MultipartEncoder
mp_encoder = MultipartEncoder(
fields={
'foo': 'bar',
# plain file object, no filename or mime type produces a
# Content-Disposition header with just the part name
'spam': ('spam.txt', open('spam.txt', 'rb'), 'text/plain'),
}
)
r = requests.post(
'http://httpbin.org/post',
data=mp_encoder, # The MultipartEncoder is posted as data, don't use files=...!
# The MultipartEncoder provides the content-type header with the boundary:
headers={'Content-Type': mp_encoder.content_type}
)
Fields follow the same conventions; use a tuple with between 2 and 4 elements to add a filename, part mime-type or extra headers. Unlike the files parameter, no attempt is made to find a default filename value if you don't use a tuple.
Requests has changed since some of the previous answers were written. Have a look at this Issue on Github for more details and this comment for an example.
In short, the files parameter takes a dictionary with the key being the name of the form field and the value being either a string or a 2, 3 or 4-length tuple, as described in the section POST a Multipart-Encoded File in the Requests quickstart:
>>> url = 'http://httpbin.org/post'
>>> files = {'file': ('report.xls', open('report.xls', 'rb'), 'application/vnd.ms-excel', {'Expires': '0'})}
In the above, the tuple is composed as follows:
(filename, data, content_type, headers)
If the value is just a string, the filename will be the same as the key, as in the following:
>>> files = {'obvius_session_id': '72c2b6f406cdabd578c5fd7598557c52'}
Content-Disposition: form-data; name="obvius_session_id"; filename="obvius_session_id"
Content-Type: application/octet-stream
72c2b6f406cdabd578c5fd7598557c52
If the value is a tuple and the first entry is None the filename property will not be included:
>>> files = {'obvius_session_id': (None, '72c2b6f406cdabd578c5fd7598557c52')}
Content-Disposition: form-data; name="obvius_session_id"
Content-Type: application/octet-stream
72c2b6f406cdabd578c5fd7598557c52
You need to use the files parameter to send a multipart form POST request even when you do not need to upload any files.
From the original requests source:
def request(method, url, **kwargs):
"""Constructs and sends a :class:`Request <Request>`.
...
:param files: (optional) Dictionary of ``'name': file-like-objects``
(or ``{'name': file-tuple}``) for multipart encoding upload.
``file-tuple`` can be a 2-tuple ``('filename', fileobj)``,
3-tuple ``('filename', fileobj, 'content_type')``
or a 4-tuple ``('filename', fileobj, 'content_type', custom_headers)``,
where ``'content-type'`` is a string
defining the content type of the given file
and ``custom_headers`` a dict-like object
containing additional headers to add for the file.
The relevant part is: file-tuple can be a:
2-tuple (filename, fileobj)
3-tuple (filename, fileobj, content_type)
4-tuple (filename, fileobj, content_type, custom_headers).
☝ What might not be obvious is that fileobj can be either an actual file object when dealing with files, OR a string when dealing with plain text fields.
Based on the above, the simplest multipart form request that includes both files to upload and form fields will look like this:
import requests
multipart_form_data = {
'upload': ('custom_file_name.zip', open('myfile.zip', 'rb')),
'action': (None, 'store'),
'path': (None, '/path1')
}
response = requests.post('https://httpbin.org/post', files=multipart_form_data)
print(response.content)
☝ Note the None as the first argument in the tuple for plain text fields — this is a placeholder for the filename field which is only used for file uploads, but for text fields passing None as the first parameter is required in order for the data to be submitted.
Multiple fields with the same name
If you need to post multiple fields with the same name then instead of a dictionary you can define your payload as a list (or a tuple) of tuples:
multipart_form_data = (
('file2', ('custom_file_name.zip', open('myfile.zip', 'rb'))),
('action', (None, 'store')),
('path', (None, '/path1')),
('path', (None, '/path2')),
('path', (None, '/path3')),
)
Streaming requests API
If the above API is not pythonic enough for you, then consider using requests toolbelt (pip install requests_toolbelt) which is an extension of the core requests module that provides support for file upload streaming as well as the MultipartEncoder which can be used instead of files, and which also lets you define the payload as a dictionary, tuple or list.
MultipartEncoder can be used both for multipart requests with or without actual upload fields. It must be assigned to the data parameter.
import requests
from requests_toolbelt.multipart.encoder import MultipartEncoder
multipart_data = MultipartEncoder(
fields={
# a file upload field
'file': ('file.zip', open('file.zip', 'rb'), 'text/plain')
# plain text fields
'field0': 'value0',
'field1': 'value1',
}
)
response = requests.post('http://httpbin.org/post', data=multipart_data,
headers={'Content-Type': multipart_data.content_type})
If you need to send multiple fields with the same name, or if the order of form fields is important, then a tuple or a list can be used instead of a dictionary:
multipart_data = MultipartEncoder(
fields=(
('action', 'ingest'),
('item', 'spam'),
('item', 'sausage'),
('item', 'eggs'),
)
)
Here is the simple code snippet to upload a single file with additional parameters using requests:
url = 'https://<file_upload_url>'
fp = '/Users/jainik/Desktop/data.csv'
files = {'file': open(fp, 'rb')}
payload = {'file_id': '1234'}
response = requests.put(url, files=files, data=payload, verify=False)
Please note that you don't need to explicitly specify any content type.
NOTE: Wanted to comment on one of the above answers but could not because of low reputation so drafted a new response here.
By specifying a files parameter in the POST request, the Content-Type of the request is automatically set to multipart/form-data (followed by the boundary string used to separate each body part in the multipart payload), whether you send only files, or form-data and files at the same time (thus, one shouldn't attempt setting the Content-Type manually in this case). Whereas, if only form-data were sent, the Content-Type would automatically be set to application/x-www-form-urlencoded.
You can print out the Content-Type header of the request to verify the above using the example given below, which shows how to upload multiple files (or a single file) with (optionally) the same key (i.e., 'files' in the case below), as well as with optional form-data (i.e., data=data in the example below). The documentation on how to POST single and multiple files can be found here and here, respectively. In case you need to upload large files without reading them into memory, have a look at Streaming Uploads.
For the server side—in case this is needed—please have a look at this answer, from which the code snippet below has been taken, and which uses the FastAPI web framework.
import requests
url = 'http://127.0.0.1:8000/submit'
files = [('files', open('a.txt', 'rb')), ('files', open('b.txt', 'rb'))]
#file = {'file': open('a.txt','rb')} # to send a single file
data ={"name": "foo", "point": 0.13, "is_accepted": False}
r = requests.post(url=url, data=data, files=files)
print(r.json())
print(r.request.headers['content-type'])
You need to use the name attribute of the upload file that is in the HTML of the site. Example:
autocomplete="off" name="image">
You see name="image">? You can find it in the HTML of a site for uploading the file. You need to use it to upload the file with Multipart/form-data
script:
import requests
site = 'https://prnt.sc/upload.php' # the site where you upload the file
filename = 'image.jpg' # name example
Here, in the place of image, add the name of the upload file in HTML
up = {'image':(filename, open(filename, 'rb'), "multipart/form-data")}
If the upload requires to click the button for upload, you can use like that:
data = {
"Button" : "Submit",
}
Then start the request
request = requests.post(site, files=up, data=data)
And done, file uploaded succesfully
import requests
# assume sending two files
url = "put ur url here"
f1 = open("file 1 path", 'rb')
f2 = open("file 2 path", 'rb')
response = requests.post(url,files={"file1 name": f1, "file2 name":f2})
print(response)
To clarify examples given above,
"You need to use the files parameter to send a multipart form POST request even when you do not need to upload any files."
files={}
won't work, unfortunately.
You will need to put some dummy values in, e.g.
files={"foo": "bar"}
I came up against this when trying to upload files to Bitbucket's REST API and had to write this abomination to avoid the dreaded "Unsupported Media Type" error:
url = "https://my-bitbucket.com/rest/api/latest/projects/FOO/repos/bar/browse/foobar.txt"
payload = {'branch': 'master',
'content': 'text that will appear in my file',
'message': 'uploading directly from python'}
files = {"foo": "bar"}
response = requests.put(url, data=payload, files=files)
:O=
Send multipart/form-data key and value
curl command:
curl -X PUT http://127.0.0.1:8080/api/xxx ...
-H 'content-type: multipart/form-data; boundary=----xxx' \
-F taskStatus=1
python requests - More complicated POST requests:
updateTaskUrl = "http://127.0.0.1:8080/api/xxx"
updateInfoDict = {
"taskStatus": 1,
}
resp = requests.put(updateTaskUrl, data=updateInfoDict)
Send multipart/form-data file
curl command:
curl -X POST http://127.0.0.1:8080/api/xxx ...
-H 'content-type: multipart/form-data; boundary=----xxx' \
-F file=#/Users/xxx.txt
python requests - POST a Multipart-Encoded File:
filePath = "/Users/xxx.txt"
fileFp = open(filePath, 'rb')
fileInfoDict = {
"file": fileFp,
}
resp = requests.post(uploadResultUrl, files=fileInfoDict)
that's all.
import json
import os
import requests
from requests_toolbelt import MultipartEncoder
AUTH_API_ENDPOINT = "http://localhost:3095/api/auth/login"
def file_upload(path_img, token ):
url = 'http://localhost:3095/api/shopping/product/image'
name_img = os.path.basename(path_img)
mp_encoder = MultipartEncoder(
fields={
'email': 'mcm9#gmail.com',
'source': 'tmall',
'productId': 'product_0001',
'image': (name_img, open(path_img, 'rb'), 'multipart/form-data')
#'spam': ('spam.txt', open('spam.txt', 'rb'), 'text/plain'),
}
)
head = {'Authorization': 'Bearer {}'.format(token),
'Content-Type': mp_encoder.content_type}
with requests.Session() as s:
result = s.post(url, data=mp_encoder, headers=head)
return result
def do_auth(username, password, url=AUTH_API_ENDPOINT):
data = {
"email": username,
"password": password
}
# sending post request and saving response as response object
r = requests.post(url=url, data=data)
# extracting response text
response_text = r.text
d = json.loads(response_text)
# print(d)
return d
if __name__ == '__main__':
result = do_auth('mcm4#gmail.com','123456')
token = result.get('data').get('payload').get('token')
print(token)
result = file_upload('/home/mcm/Pictures/1234.png',token)
print(result.json())
Here is the python snippet you need to upload one large single file as multipart formdata. With NodeJs Multer middleware running on the server side.
import requests
latest_file = 'path/to/file'
url = "http://httpbin.org/apiToUpload"
files = {'fieldName': open(latest_file, 'rb')}
r = requests.put(url, files=files)
For the server side please check the multer documentation at: https://github.com/expressjs/multer
here the field single('fieldName') is used to accept one single file, as in:
var upload = multer().single('fieldName');
This is one way to send file in multipart request
import requests
headers = {"Authorization": "Bearer <token>"}
myfile = 'file.txt'
myfile2 = {'file': (myfile, open(myfile, 'rb'),'application/octet-stream')}
url = 'https://example.com/path'
r = requests.post(url, files=myfile2, headers=headers,verify=False)
print(r.content)
Other approach
import requests
url = "https://example.com/path"
payload={}
files=[
('file',('file',open('/path/to/file','rb'),'application/octet-stream'))
]
headers = {
'Authorization': 'Bearer <token>'
}
response = requests.request("POST", url, headers=headers, data=payload, files=files)
print(response.text)
I have tested both , both works fine.
I'm trying to send a request to URL_server with request module in python 3.
This works for me:
# -*- coding: utf-8 *-*
import json, requests
URL_SERVER_TO_POST_DATA = "URL_to_send_POST_request"
HEADERS = {"Content-Type" : "multipart/form-data;"}
def getPointsCC_Function():
file_data = {
'var1': (None, "valueOfYourVariable_1"),
'var2': (None, "valueOfYourVariable_2")
}
try:
resElastic = requests.post(URL_GET_BALANCE, files=file_data)
res = resElastic.json()
except Exception as e:
print(e)
print (json.dumps(res, indent=4, sort_keys=True))
getPointsCC_Function()
Where:
URL_SERVER_TO_POST_DATA = Server where we going to send data
HEADERS = Headers sended
file_data = Params sended
Postman generated code for file upload with additional form fields:
import http.client
import mimetypes
from codecs import encode
conn = http.client.HTTPSConnection("data.XXXX.com")
dataList = []
boundary = 'wL36Yn8afVp8Ag7AmP8qZ0SA4n1v9T'
dataList.append(encode('--' + boundary))
dataList.append(encode('Content-Disposition: form-data; name=batchSize;'))
dataList.append(encode('Content-Type: {}'.format('text/plain')))
dataList.append(encode(''))
dataList.append(encode("1"))
dataList.append(encode('--' + boundary))
dataList.append(encode('Content-Disposition: form-data; name=file; filename={0}'.format('FileName-1.json')))
fileType = mimetypes.guess_type('FileName-1.json')[0] or 'application/octet-stream'
dataList.append(encode('Content-Type: {}'.format(fileType)))
dataList.append(encode(''))
with open('FileName-1.json', 'rb') as f:
dataList.append(f.read())
dataList.append(encode('--'+boundary+'--'))
dataList.append(encode(''))
body = b'\r\n'.join(dataList)
payload = body
headers = {
'Cookie': 'XXXXXXXXXXX',
'Content-type': 'multipart/form-data; boundary={}'.format(boundary)
}
conn.request("POST", "/fileupload/uri/XXXX", payload, headers)
res = conn.getresponse()
data = res.read()
print(data.decode("utf-8"))
Related
I'm trying to send a POST API call using requests in Python, but I'm running into a problem with how to pass both json data and multipart file at the same time. I can get it to work in Insomnia, but then recreating that in a Python module with variables that I am populating is difficult.
The API documentation says there are two parameters -- file and metadata. metadata is a json object that includes keys including matter, title, and description. matter is required. All of the options that I've looked at result with a response that the matter parameter is missing.
I've tried using requests_toolkit and it's multipart encoder:
matter_id = 5
payload_data = {}
payload_data['matter'] = matter_id
mp_encoder = MultipartEncoder(fields={'metadata': "{" + str(payload_data) + "}", 'file': (None, open(file_path, 'rb'), 'text/plain') })
response = requests.post(url, data=mp_encoder, headers={'Content-Type': mp_encoder.content_type}, auth=(apiuser, apipassword))
If I didn't use str(payload_data), I got errors about encoding. I tried adding the extra {} but that didn't seem to make that happen either.
I've tried with just requests:
matter_id = 5
payload_data = {}
payload_data['matter'] = matter_id
file_content = {'file': open(file_path, 'rb')}
payload = {'metadata': payload_data}
response = requests.post(url, data=payload, files=file_content, auth=(apiuser, apipassword))
I also tried payload=payload, params=payload instead of data=payload and no luck.
In the above example, I tried combining the payload and file_content into one dictionary to pass in the files parameter:
payloadv2 = {'file': (None, open(file_path, 'rb')), 'metadata': (None, {'matter': matter_id})}
And that also didn't work.
You should convert python dict to JSON string representation using json.dumps(payload_data) instead of concatenating "{" and "}" since the latter might be error-prone.
I'm performing a simple task of uploading a file using Python requests library. I searched Stack Overflow and no one seemed to have the same problem, namely, that the file is not received by the server:
import requests
url='http://nesssi.cacr.caltech.edu/cgi-bin/getmulticonedb_release2.cgi/post'
files={'files': open('file.txt','rb')}
values={'upload_file' : 'file.txt' , 'DB':'photcat' , 'OUT':'csv' , 'SHORT':'short'}
r=requests.post(url,files=files,data=values)
I'm filling the value of 'upload_file' keyword with my filename, because if I leave it blank, it says
Error - You must select a file to upload!
And now I get
File file.txt of size bytes is uploaded successfully!
Query service results: There were 0 lines.
Which comes up only if the file is empty. So I'm stuck as to how to send my file successfully. I know that the file works because if I go to this website and manually fill in the form it returns a nice list of matched objects, which is what I'm after. I'd really appreciate all hints.
Some other threads related (but not answering my problem):
Send file using POST from a Python script
http://docs.python-requests.org/en/latest/user/quickstart/#response-content
Uploading files using requests and send extra data
http://docs.python-requests.org/en/latest/user/advanced/#body-content-workflow
If upload_file is meant to be the file, use:
files = {'upload_file': open('file.txt','rb')}
values = {'DB': 'photcat', 'OUT': 'csv', 'SHORT': 'short'}
r = requests.post(url, files=files, data=values)
and requests will send a multi-part form POST body with the upload_file field set to the contents of the file.txt file.
The filename will be included in the mime header for the specific field:
>>> import requests
>>> open('file.txt', 'wb') # create an empty demo file
<_io.BufferedWriter name='file.txt'>
>>> files = {'upload_file': open('file.txt', 'rb')}
>>> print(requests.Request('POST', 'http://example.com', files=files).prepare().body.decode('ascii'))
--c226ce13d09842658ffbd31e0563c6bd
Content-Disposition: form-data; name="upload_file"; filename="file.txt"
--c226ce13d09842658ffbd31e0563c6bd--
Note the filename="file.txt" parameter.
You can use a tuple for the files mapping value, with between 2 and 4 elements, if you need more control. The first element is the filename, followed by the contents, and an optional content-type header value and an optional mapping of additional headers:
files = {'upload_file': ('foobar.txt', open('file.txt','rb'), 'text/x-spam')}
This sets an alternative filename and content type, leaving out the optional headers.
If you are meaning the whole POST body to be taken from a file (with no other fields specified), then don't use the files parameter, just post the file directly as data. You then may want to set a Content-Type header too, as none will be set otherwise. See Python requests - POST data from a file.
(2018) the new python requests library has simplified this process, we can use the 'files' variable to signal that we want to upload a multipart-encoded file
url = 'http://httpbin.org/post'
files = {'file': open('report.xls', 'rb')}
r = requests.post(url, files=files)
r.text
Client Upload
If you want to upload a single file with Python requests library, then requests lib supports streaming uploads, which allow you to send large files or streams without reading into memory.
with open('massive-body', 'rb') as f:
requests.post('http://some.url/streamed', data=f)
Server Side
Then store the file on the server.py side such that save the stream into file without loading into the memory. Following is an example with using Flask file uploads.
#app.route("/upload", methods=['POST'])
def upload_file():
from werkzeug.datastructures import FileStorage
FileStorage(request.stream).save(os.path.join(app.config['UPLOAD_FOLDER'], filename))
return 'OK', 200
Or use werkzeug Form Data Parsing as mentioned in a fix for the issue of "large file uploads eating up memory" in order to avoid using memory inefficiently on large files upload (s.t. 22 GiB file in ~60 seconds. Memory usage is constant at about 13 MiB.).
#app.route("/upload", methods=['POST'])
def upload_file():
def custom_stream_factory(total_content_length, filename, content_type, content_length=None):
import tempfile
tmpfile = tempfile.NamedTemporaryFile('wb+', prefix='flaskapp', suffix='.nc')
app.logger.info("start receiving file ... filename => " + str(tmpfile.name))
return tmpfile
import werkzeug, flask
stream, form, files = werkzeug.formparser.parse_form_data(flask.request.environ, stream_factory=custom_stream_factory)
for fil in files.values():
app.logger.info(" ".join(["saved form name", fil.name, "submitted as", fil.filename, "to temporary file", fil.stream.name]))
# Do whatever with stored file at `fil.stream.name`
return 'OK', 200
You can send any file via post api while calling the API just need to mention files={'any_key': fobj}
import requests
import json
url = "https://request-url.com"
headers = {"Content-Type": "application/json; charset=utf-8"}
with open(filepath, 'rb') as fobj:
response = requests.post(url, headers=headers, files={'file': fobj})
print("Status Code", response.status_code)
print("JSON Response ", response.json())
#martijn-pieters answer is correct, however I wanted to add a bit of context to data= and also to the other side, in the Flask server, in the case where you are trying to upload files and a JSON.
From the request side, this works as Martijn describes:
files = {'upload_file': open('file.txt','rb')}
values = {'DB': 'photcat', 'OUT': 'csv', 'SHORT': 'short'}
r = requests.post(url, files=files, data=values)
However, on the Flask side (the receiving webserver on the other side of this POST), I had to use form
#app.route("/sftp-upload", methods=["POST"])
def upload_file():
if request.method == "POST":
# the mimetype here isnt application/json
# see here: https://stackoverflow.com/questions/20001229/how-to-get-posted-json-in-flask
body = request.form
print(body) # <- immutable dict
body = request.get_json() will return nothing. body = request.get_data() will return a blob containing lots of things like the filename etc.
Here's the bad part: on the client side, changing data={} to json={} results in this server not being able to read the KV pairs! As in, this will result in a {} body above:
r = requests.post(url, files=files, json=values). # No!
This is bad because the server does not have control over how the user formats the request; and json= is going to be the habbit of requests users.
Upload:
with open('file.txt', 'rb') as f:
files = {'upload_file': f.read()}
values = {'DB': 'photcat', 'OUT': 'csv', 'SHORT': 'short'}
r = requests.post(url, files=files, data=values)
Download (Django):
with open('file.txt', 'wb') as f:
f.write(request.FILES['upload_file'].file.read())
Regarding the answers given so far, there was always something missing that prevented it to work on my side. So let me show you what worked for me:
import json
import os
import requests
API_ENDPOINT = "http://localhost:80"
access_token = "sdfJHKsdfjJKHKJsdfJKHJKysdfJKHsdfJKHs" # TODO: get fresh Token here
def upload_engagement_file(filepath):
url = API_ENDPOINT + "/api/files" # add any URL parameters if needed
hdr = {"Authorization": "Bearer %s" % access_token}
with open(filepath, "rb") as fobj:
file_obj = fobj.read()
file_basename = os.path.basename(filepath)
file_to_upload = {"file": (str(file_basename), file_obj)}
finfo = {"fullPath": filepath}
upload_response = requests.post(url, headers=hdr, files=file_to_upload, data=finfo)
fobj.close()
# print("Status Code ", upload_response.status_code)
# print("JSON Response ", upload_response.json())
return upload_response
Note that requests.post(...) needs
a url parameter, containing the full URL of the API endpoint you're calling, using the API_ENDPOINT, assuming we have an http://localhost:8000/api/files endpoint to POST a file
a headers parameter, containing at least the authorization (bearer token)
a files parameter taking the name of the file plus the entire file content
a data parameter taking just the path and file name
Installation required (console):
pip install requests
What you get back from the function call is a response object containing a status code and also the full error message in JSON format. The commented print statements at the end of upload_engagement_file are showing you how you can access them.
Note: Some useful additional information about the requests library can be found here
Some may need to upload via a put request and this is slightly different that posting data. It is important to understand how the server expects the data in order to form a valid request. A frequent source of confusion is sending multipart-form data when it isn't accepted. This example uses basic auth and updates an image via a put request.
url = 'foobar.com/api/image-1'
basic = requests.auth.HTTPBasicAuth('someuser', 'password123')
# Setting the appropriate header is important and will vary based
# on what you upload
headers = {'Content-Type': 'image/png'}
with open('image-1.png', 'rb') as img_1:
r = requests.put(url, auth=basic, data=img_1, headers=headers)
While the requests library makes working with http requests a lot easier, some of its magic and convenience obscures just how to craft more nuanced requests.
In Ubuntu you can apply this way,
to save file at some location (temporary) and then open and send it to API
path = default_storage.save('static/tmp/' + f1.name, ContentFile(f1.read()))
path12 = os.path.join(os.getcwd(), "static/tmp/" + f1.name)
data={} #can be anything u want to pass along with File
file1 = open(path12, 'rb')
header = {"Content-Disposition": "attachment; filename=" + f1.name, "Authorization": "JWT " + token}
res= requests.post(url,data,header)
I tried to sole this problem myself for a few days searching on examples and documentations, also it wasn't solved on ruSO. So, I hope that solution will be found here, on enSO.
I develop a service for automatic creation of ads at Vk social network using Python and Google App Engine. Initially, pictures for ads are loaded to my server (part 1), then they are uploaded to Vk server at some time (parts 2.1 and 2.2). It seems that pictures are loaded and stored on my server correctly (I downloaded them and compared with original ones — every byte is the same). But I attach the part 1 code just in case.
To upload a picture to Vk.Ads firstly I need to get a URL — this is simple, so skip it. Secondly, I need to send a POST request to this link with field file with binary content of the photo (API documentation). I created two ways for that (2.1 and 2.2), but both of them returns errcode: 2 which means corrupted file. To my mind, the problem is about the requests, but I don't exclude the possibility of it files uploading/storage on my server, or some strange work of the API. I'll appreciate any answers and comments.
1. Uploading to my server
import webapp2
from google.appengine.ext import ndb
# stores pictures on the server
class Photo(ndb.Model):
name = ndb.StringProperty()
img = ndb.BlobProperty()
#staticmethod
def get(name):
retval = Photo.query(Photo.name == name).get()
return retval
#staticmethod
def create(name, blob):
retval = Photo()
retval.name = name
retval.img = blob
return retval
class PhotosPage(webapp2.RequestHandler):
def get(self):
# general content of the page:
html = '''<form action="/photos" method="post" enctype="multipart/form-data">
<input type="file" name="flimg"/>
<input value="new_pic" name="flname"/>
<input type="submit" value="Upload"/> </form>'''
def post(self):
n = str(self.request.get('flname'))
f = self.request.get('flimg')
p = Photo.get(n)
if p:
p.img = f
else:
p = Photo.create(n, f)
p.put()
2.1. POST to API, approach #1, using urlfetch и poster:
from poster.encode import multipart_encode, MultipartParam
from google.appengine.api import urlfetch
name = 'file'
content = ... # file binary content
where = ... # gotten URL
options = {
'file': MultipartParam(
name=name,
value=content,
filename=name,
filetype='image/png',
filesize=len(content))
}
data, headers = multipart_encode(options)
pocket = "".join(data)
result = urlfetch.fetch(
url=where,
payload=pocket,
method=urlfetch.POST,
headers=headers)
2.2. POST to API, approach #2, using requests:
import requests
name = 'file'
content = ... # file binary content
where = ... # gotten URL
# I also tried without this dict; is it necessary?
data = {
'fileName': name,
'fileSize': len(content),
'description': 'undefined',
}
result = requests.post(where, files={name: StringIO(content)}, data=data)
In addition, for the second approach I extracted the content of my request:
POST
https://pu.vk.com/c.../upload.php?act=ads_add&mid=...&size=m&rdsn=1&hash_time=...&hash=...&rhash=...&api=1
Content-Length: 15946
Content-Type: multipart/form-data; boundary=b4b260eace4e4a7082a99753b74cf51f
--b4b260eace4e4a7082a99753b74cf51f
Content-Disposition: form-data; name="description"
undefined
--b4b260eace4e4a7082a99753b74cf51f
Content-Disposition: form-data; name="fileSize"
15518
--b4b260eace4e4a7082a99753b74cf51f
Content-Disposition: form-data; name="fileName"
file
--b4b260eace4e4a7082a99753b74cf51f
Content-Disposition: form-data; name="file"; filename="file"
< File binary content >
--b4b260eace4e4a7082a99753b74cf51f--
UPDATE.
Thanks to SwiftStudier, I found the origin of the problem: StringIO and BytesIO don't behave identical to file open. If I use just open the code works well, but it doesn't with virtual file. How it can be solved?
import requests
from io import BytesIO
with open('path.to/file.png', 'rb') as fin:
content = BytesIO(fin.read())
token = '...'
url = 'https://api.vk.com/method/ads.getUploadURL?access_token=' + token + '&ad_format=2'
upload_url = requests.get(url).json()['response']
post_fields = {
'access_token': token
}
data_fields = {
# This works:
# 'file': open('path.to/file.png', 'rb')
# But this does not:
'file': content
}
response = requests.post(upload_url, data=post_fields, files=data_fields)
print(response.text)
After a lot of experiments and investigating of different HTTP requests content I found out the only difference between wrong and working code. It was about 4 bytes only: file name MUST contain the extension. Vk API even ignores Content-Type: image/png, but needs .png or similar in filename. So, this doesn't work:
requests.post(upload_url, files={
'file': BytesIO('<binary file content>')
})
But this option works properly:
requests.post(upload_url, files={
'file': ('file.png', BytesIO('<binary file content>'), 'image/png')
})
Just like this one, which is not available for GAE:
requests.post(upload_url, files={
'file': open('/path/to/image.png', 'rb')
})
Both StringIO and StringIO are appropriate for that task. As mentioned, Content-Type does not matter, it can be just multipart/form-data.
Not sure if it can help, but I'll post it anyway
I used requests to upload an image to ads
import requests
token = '***'
url = f'https://api.vk.com/method/ads.getUploadURL?access_token={token}&ad_format=1' # I set add_format randomly just to avoid an error of this parameter was missing
upload_url = requests.get(url).json()['response']
post_fields = {
'access_token': token
}
data_fields = {
'file': open('/path/to/image.png', 'rb')
}
response = requests.post(upload_url, data=post_fields, files=data_fields)
print(response.text)
The result looks like valid photo upload, the data received can be used in further actions with ad API.
I'm performing a simple task of uploading a file using Python requests library. I searched Stack Overflow and no one seemed to have the same problem, namely, that the file is not received by the server:
import requests
url='http://nesssi.cacr.caltech.edu/cgi-bin/getmulticonedb_release2.cgi/post'
files={'files': open('file.txt','rb')}
values={'upload_file' : 'file.txt' , 'DB':'photcat' , 'OUT':'csv' , 'SHORT':'short'}
r=requests.post(url,files=files,data=values)
I'm filling the value of 'upload_file' keyword with my filename, because if I leave it blank, it says
Error - You must select a file to upload!
And now I get
File file.txt of size bytes is uploaded successfully!
Query service results: There were 0 lines.
Which comes up only if the file is empty. So I'm stuck as to how to send my file successfully. I know that the file works because if I go to this website and manually fill in the form it returns a nice list of matched objects, which is what I'm after. I'd really appreciate all hints.
Some other threads related (but not answering my problem):
Send file using POST from a Python script
http://docs.python-requests.org/en/latest/user/quickstart/#response-content
Uploading files using requests and send extra data
http://docs.python-requests.org/en/latest/user/advanced/#body-content-workflow
If upload_file is meant to be the file, use:
files = {'upload_file': open('file.txt','rb')}
values = {'DB': 'photcat', 'OUT': 'csv', 'SHORT': 'short'}
r = requests.post(url, files=files, data=values)
and requests will send a multi-part form POST body with the upload_file field set to the contents of the file.txt file.
The filename will be included in the mime header for the specific field:
>>> import requests
>>> open('file.txt', 'wb') # create an empty demo file
<_io.BufferedWriter name='file.txt'>
>>> files = {'upload_file': open('file.txt', 'rb')}
>>> print(requests.Request('POST', 'http://example.com', files=files).prepare().body.decode('ascii'))
--c226ce13d09842658ffbd31e0563c6bd
Content-Disposition: form-data; name="upload_file"; filename="file.txt"
--c226ce13d09842658ffbd31e0563c6bd--
Note the filename="file.txt" parameter.
You can use a tuple for the files mapping value, with between 2 and 4 elements, if you need more control. The first element is the filename, followed by the contents, and an optional content-type header value and an optional mapping of additional headers:
files = {'upload_file': ('foobar.txt', open('file.txt','rb'), 'text/x-spam')}
This sets an alternative filename and content type, leaving out the optional headers.
If you are meaning the whole POST body to be taken from a file (with no other fields specified), then don't use the files parameter, just post the file directly as data. You then may want to set a Content-Type header too, as none will be set otherwise. See Python requests - POST data from a file.
(2018) the new python requests library has simplified this process, we can use the 'files' variable to signal that we want to upload a multipart-encoded file
url = 'http://httpbin.org/post'
files = {'file': open('report.xls', 'rb')}
r = requests.post(url, files=files)
r.text
Client Upload
If you want to upload a single file with Python requests library, then requests lib supports streaming uploads, which allow you to send large files or streams without reading into memory.
with open('massive-body', 'rb') as f:
requests.post('http://some.url/streamed', data=f)
Server Side
Then store the file on the server.py side such that save the stream into file without loading into the memory. Following is an example with using Flask file uploads.
#app.route("/upload", methods=['POST'])
def upload_file():
from werkzeug.datastructures import FileStorage
FileStorage(request.stream).save(os.path.join(app.config['UPLOAD_FOLDER'], filename))
return 'OK', 200
Or use werkzeug Form Data Parsing as mentioned in a fix for the issue of "large file uploads eating up memory" in order to avoid using memory inefficiently on large files upload (s.t. 22 GiB file in ~60 seconds. Memory usage is constant at about 13 MiB.).
#app.route("/upload", methods=['POST'])
def upload_file():
def custom_stream_factory(total_content_length, filename, content_type, content_length=None):
import tempfile
tmpfile = tempfile.NamedTemporaryFile('wb+', prefix='flaskapp', suffix='.nc')
app.logger.info("start receiving file ... filename => " + str(tmpfile.name))
return tmpfile
import werkzeug, flask
stream, form, files = werkzeug.formparser.parse_form_data(flask.request.environ, stream_factory=custom_stream_factory)
for fil in files.values():
app.logger.info(" ".join(["saved form name", fil.name, "submitted as", fil.filename, "to temporary file", fil.stream.name]))
# Do whatever with stored file at `fil.stream.name`
return 'OK', 200
You can send any file via post api while calling the API just need to mention files={'any_key': fobj}
import requests
import json
url = "https://request-url.com"
headers = {"Content-Type": "application/json; charset=utf-8"}
with open(filepath, 'rb') as fobj:
response = requests.post(url, headers=headers, files={'file': fobj})
print("Status Code", response.status_code)
print("JSON Response ", response.json())
#martijn-pieters answer is correct, however I wanted to add a bit of context to data= and also to the other side, in the Flask server, in the case where you are trying to upload files and a JSON.
From the request side, this works as Martijn describes:
files = {'upload_file': open('file.txt','rb')}
values = {'DB': 'photcat', 'OUT': 'csv', 'SHORT': 'short'}
r = requests.post(url, files=files, data=values)
However, on the Flask side (the receiving webserver on the other side of this POST), I had to use form
#app.route("/sftp-upload", methods=["POST"])
def upload_file():
if request.method == "POST":
# the mimetype here isnt application/json
# see here: https://stackoverflow.com/questions/20001229/how-to-get-posted-json-in-flask
body = request.form
print(body) # <- immutable dict
body = request.get_json() will return nothing. body = request.get_data() will return a blob containing lots of things like the filename etc.
Here's the bad part: on the client side, changing data={} to json={} results in this server not being able to read the KV pairs! As in, this will result in a {} body above:
r = requests.post(url, files=files, json=values). # No!
This is bad because the server does not have control over how the user formats the request; and json= is going to be the habbit of requests users.
Upload:
with open('file.txt', 'rb') as f:
files = {'upload_file': f.read()}
values = {'DB': 'photcat', 'OUT': 'csv', 'SHORT': 'short'}
r = requests.post(url, files=files, data=values)
Download (Django):
with open('file.txt', 'wb') as f:
f.write(request.FILES['upload_file'].file.read())
Regarding the answers given so far, there was always something missing that prevented it to work on my side. So let me show you what worked for me:
import json
import os
import requests
API_ENDPOINT = "http://localhost:80"
access_token = "sdfJHKsdfjJKHKJsdfJKHJKysdfJKHsdfJKHs" # TODO: get fresh Token here
def upload_engagement_file(filepath):
url = API_ENDPOINT + "/api/files" # add any URL parameters if needed
hdr = {"Authorization": "Bearer %s" % access_token}
with open(filepath, "rb") as fobj:
file_obj = fobj.read()
file_basename = os.path.basename(filepath)
file_to_upload = {"file": (str(file_basename), file_obj)}
finfo = {"fullPath": filepath}
upload_response = requests.post(url, headers=hdr, files=file_to_upload, data=finfo)
fobj.close()
# print("Status Code ", upload_response.status_code)
# print("JSON Response ", upload_response.json())
return upload_response
Note that requests.post(...) needs
a url parameter, containing the full URL of the API endpoint you're calling, using the API_ENDPOINT, assuming we have an http://localhost:8000/api/files endpoint to POST a file
a headers parameter, containing at least the authorization (bearer token)
a files parameter taking the name of the file plus the entire file content
a data parameter taking just the path and file name
Installation required (console):
pip install requests
What you get back from the function call is a response object containing a status code and also the full error message in JSON format. The commented print statements at the end of upload_engagement_file are showing you how you can access them.
Note: Some useful additional information about the requests library can be found here
Some may need to upload via a put request and this is slightly different that posting data. It is important to understand how the server expects the data in order to form a valid request. A frequent source of confusion is sending multipart-form data when it isn't accepted. This example uses basic auth and updates an image via a put request.
url = 'foobar.com/api/image-1'
basic = requests.auth.HTTPBasicAuth('someuser', 'password123')
# Setting the appropriate header is important and will vary based
# on what you upload
headers = {'Content-Type': 'image/png'}
with open('image-1.png', 'rb') as img_1:
r = requests.put(url, auth=basic, data=img_1, headers=headers)
While the requests library makes working with http requests a lot easier, some of its magic and convenience obscures just how to craft more nuanced requests.
In Ubuntu you can apply this way,
to save file at some location (temporary) and then open and send it to API
path = default_storage.save('static/tmp/' + f1.name, ContentFile(f1.read()))
path12 = os.path.join(os.getcwd(), "static/tmp/" + f1.name)
data={} #can be anything u want to pass along with File
file1 = open(path12, 'rb')
header = {"Content-Disposition": "attachment; filename=" + f1.name, "Authorization": "JWT " + token}
res= requests.post(url,data,header)
I want to request to my server running in python flask with file and some meta information. Hence my request content-Type will be 'multipart/form-data. Is there a way i can set the content type of file like image/jpg, image/gif etc...
How do i set the content-type for the file. Is it possible or not
If you make each file specification a tuple, you can specify the mime type as a third parameter:
files = {
'file1': ('foo.gif', open('foo.gif', 'rb'), 'image/gif'),
'file2': ('bar.png', open('bar.png', 'rb'), 'image/png'),
}
response = requests.post(url, files=files)
You can give a 4th parameter as well, which must be a dictionary with additional headers for each part.
See the Requests API documentation:
file-tuple can be a 2-tuple ('filename', fileobj), 3-tuple ('filename', fileobj, 'content_type') or a 4-tuple ('filename', fileobj, 'content_type', custom_headers), where 'content-type' is a string defining the content type of the given file and custom_headers a dict-like object containing additional headers to add for the file.
reference
import requests
url = "http://png_upload_example/upload"
# files = [(<key>, (<filename>, open(<file location>, 'rb'), <content type>))]
files = [('upload', ('thumbnail.png', open('thumbnail.png', 'rb'), 'image/png'))]
response = requests.request("POST", url, files = files)