Is there a way to stream a file directly to the filesystem? Even if the connection is lost, I want to see all contents in that specific file in the fs. Just like wget or curl.
However, using request leads to the issue of first downloading the content of a response and then writing it to a filesystem.
with open(file_name, "wb") as file:
response = get(url) # may take some time
file.write(response.content)
Problem: while the file is "downloading" it is stored elsewhere (I guess in memory or a temporarily splace in the filesystem). That means I have a 0 byte file as long as the request is not (successfully) finished.
Can I solve this problem without using a third party lib?
Streaming directly to file can be achieved with requests and stream=true, or see more useful examples
with open(file_name, 'wb') as f:
with requests.get(url, stream=True) as r:
shutil.copyfileobj(r.raw, f)
Related
i'm trying to POST upload large files with multipart/form-data encoded, but i want to use requests module only, not using requests_toolbelt.
is there anyway i can upload the file by chunking it to reduce memory usage?
here's my current code
with open(file, 'rb') as f:
r = requests.post('url', files={'report.mp4': f})
print(r.text)
the reason i don't want to use requests_toolbelt, is that when i request the POST the code finishes running, it doesn't give any error, it just finishes and the request text is empty
here's the code that i'm using that fails.
m = MultipartEncoder(
fields={'file': ('filename', open(file, 'rb'), 'video/mp4')}
)
r = requests.post(url, data=m,
headers={'Content-Type': m.content_type})
What I am trying to accomplish is to download a specific portion of a video file using python. Sort of what a browser will do when playing a video. If the file is 1000 Bytes, I want to download from byte 200 to 700. I know that I can download the file in parts using the method below:
file_ = open(filename, 'wb')
res = requests.get(url, stream=True)
for chunk in res.iter_content(amount):
file_.write(chunk)
file_.close()
How can I modify this code to accomplish that?
The server has to support this:
If Accept-Ranges is present in HTTP responses (and its value isn't
none), the server supports range requests. You can check this by
issuing a HEAD request.
If the server supports it you can request the part as
curl http://i.imgur.com/z4d4kWk.jpg -i -H "Range: bytes=0-1023"
Source: https://developer.mozilla.org/en-US/docs/Web/HTTP/Range_requests
Also take a look at this.
I am trying to download a .csv file from a url as shown below:
http://s3.amazonaws.com/bucket_name/path/to/file/filename?AWSAccessKeyId=Key&Expires=15&Signature=SomeCharacters
The URL is a response which I get from a webservice using python and I can't hard-code the aws credentials since the Signature changes every call and the url remains active for only 15 seconds. I tried the below:
s3.meta.client.download_file('bucketname', url, 'localpath')
and also
r = requests.get(url, auth=('username','password'), verify=False, stream=True)
r.raw.decode_content = True
with open("filename", 'wb') as f:
shutil.copyfileobj(r.raw, f)
But the above methods are not working when it comes to invoking the URL directly. Can someone please suggest something?
Thanks,
Milind
I am using netty to create an http server for some basic usage and file transfer. I used netty's example for File server in order to learn how netty handles file transfers, and created a python client using requests module to transfer the file. The python code is:
r = requests.get("http://10.154.196.99:8000")
LOG.debug(r.headers)
LOG.debug(r.content)
with open('reports.csv', "wb") as out_file:
shutil.copyfileobj(r.raw, out_file)
r.content prints the contents of the file transfered correctly.But reports.csv is empty. Also when going to the address from my browser file gets downloaded normally, with contents. What do you think is the the problem?
It worked but only I after I changed the code of while to this according to requests documentation for streaming files.
with open('reports.csv', "wb") as out_file:
for chunk in r.iter_content():
out_file.write(chunk)
Changing the file streamed by the server to a new one doesn't work. I can download the new file from web browser but not from requests and python.
Can you disable the auto decoding feature in requests version 1.2.3?
I've looked through the documentation and couldn't find anything, I'm currently experiencing a gzip decode error and want to manually debug the data coming through the request.
You can access the raw response like this:
resp = requests.get(url, stream=True)
resp.raw.read()
In order to use raw you need to set stream=True for the original request. Also, raw is a file-like object, and reading from response.content will affect the seek cursor. In other words: If you already (tried to) read response.content, response.raw.read() will return an empty string.
See FAQ: Encoded Data and Quickstart: raw response content in the requests documentation.
import requests
r = requests.get(url, stream=True)
with open(local_filename, 'wb') as f:
for chunk in r.raw.stream(1024, decode_content=False):
if chunk:
f.write(chunk)
This way, you will avoid automatic decompress of gzip-encoded response, and still write it to file chunk by chunk (useful for getting big files)