i'm trying to POST upload large files with multipart/form-data encoded, but i want to use requests module only, not using requests_toolbelt.
is there anyway i can upload the file by chunking it to reduce memory usage?
here's my current code
with open(file, 'rb') as f:
r = requests.post('url', files={'report.mp4': f})
print(r.text)
the reason i don't want to use requests_toolbelt, is that when i request the POST the code finishes running, it doesn't give any error, it just finishes and the request text is empty
here's the code that i'm using that fails.
m = MultipartEncoder(
fields={'file': ('filename', open(file, 'rb'), 'video/mp4')}
)
r = requests.post(url, data=m,
headers={'Content-Type': m.content_type})
Related
;TLDR
I want to send a file with requests.send() using multipart/form-data request without storing the file on a hard drive. Basically, I'm looking for an alternative for open() function for bytes object
Hello, I'm currently trying to send multipart/form-data request and pass in-memory files in it, but I can't figure out how to do that.
My app receives images from one source and sends them to another. Currently it sends get request directly to file, (e.g. requests.get('https://service.com/test.jpeg')), reads image's bytes and writes them into new file on the hard drive. The sending code that works looks like this:
def send_file(path_to_image: str)
url = get_upload_link()
data = {'photo': open(path_to_image, 'rb')}
r = requests.post(url, files=data)
send_file("test.jpeg")
The main issue I have with this approach is that I have to keep files on my hard drive. Sure, I can use my drive as some sort of a "temporary buffer" and just delete them after I no longer need these files, but I believe there's much more simple way to do that.
I want my function to receive bytes object and then send it. I actually tried doing that, but the backend doesn't accept them. Here's what I tried to do
Attempt 1
def send_file(image: bytes)
url = get_upload_link()
data = {'photo': open(image, 'rb')}
r = requests.post(url, files=data)
I get "ValueError: embedded null byte"
Attempt 2
def upload_photo(image: bytes):
url = get_upload_link()
file = BytesIO(image)
data = {'photo': file}
r = requests.post(url, files=data)
Backend server doesn't process my files correctly. It's like passing files=None, same response
I also tried:
sending the returning value of the methods: file.getbuffer() and file.read()
file.write(image) and then sending file
StringsIO object
etc.
Final notes
I noticed, that open() returns _io.BufferedReader object. I also looked for a way to construct its instance, but couldn't fund a way. Can someone help me, please?
UPD:
If anyone is interested, the receiving api is this
From the official documentation:
POST a Multipart-Encoded File
...
If you want, you can send strings to
be received as files:
url = 'https://httpbin.org/post'
files = {'file': ('report.csv', 'some,data,to,send\nanother,row,to,send\n')}
r = requests.post(url, files=files)
I know how to submit a post request with a file.
files = {'file': open('local.pdf', 'rb')}
r = requests.post(url, files=files)
Since I'm downloading the file from a response, I want to avoid writing the response.contents to my local disk ('local.pdf') before I submit the post request. Can I submit the file as a bytes object?
You can use io.BytesIO to do that.
Here is an example:
rawData = io.BytesIO(b"Some data: \x00\x01") # Change the content
files = {'file': rawData}
r = requests.post(url, files=files)
Is there a way to stream a file directly to the filesystem? Even if the connection is lost, I want to see all contents in that specific file in the fs. Just like wget or curl.
However, using request leads to the issue of first downloading the content of a response and then writing it to a filesystem.
with open(file_name, "wb") as file:
response = get(url) # may take some time
file.write(response.content)
Problem: while the file is "downloading" it is stored elsewhere (I guess in memory or a temporarily splace in the filesystem). That means I have a 0 byte file as long as the request is not (successfully) finished.
Can I solve this problem without using a third party lib?
Streaming directly to file can be achieved with requests and stream=true, or see more useful examples
with open(file_name, 'wb') as f:
with requests.get(url, stream=True) as r:
shutil.copyfileobj(r.raw, f)
How to upload file via sending a post request with a path to the file in Kotlin?
This is what I did in Python, it manage to work, it is plain and simple.
import requests
//file that I want to upload
path = 'download/special.db'
files = {'file': open(path, 'rb')}
r = requests.post('http://myurl/newfile/Upload', files=files)
print (r.text)
This code work and manage to print the response
I need to write one version for my Android app, is it possible to write something similar in Kotlin? send a post request and attach a files=files to my request and send. The problem, what is the kotlin equivalentof that? is it possible to write everything on one activity?
what should I write for files = {'file': open(path, 'rb')} in kotlin?
Search online, most of the method is use Retrofit, which is too complicated.
I am trying to download a .csv file from a url as shown below:
http://s3.amazonaws.com/bucket_name/path/to/file/filename?AWSAccessKeyId=Key&Expires=15&Signature=SomeCharacters
The URL is a response which I get from a webservice using python and I can't hard-code the aws credentials since the Signature changes every call and the url remains active for only 15 seconds. I tried the below:
s3.meta.client.download_file('bucketname', url, 'localpath')
and also
r = requests.get(url, auth=('username','password'), verify=False, stream=True)
r.raw.decode_content = True
with open("filename", 'wb') as f:
shutil.copyfileobj(r.raw, f)
But the above methods are not working when it comes to invoking the URL directly. Can someone please suggest something?
Thanks,
Milind