I am using netty to create an http server for some basic usage and file transfer. I used netty's example for File server in order to learn how netty handles file transfers, and created a python client using requests module to transfer the file. The python code is:
r = requests.get("http://10.154.196.99:8000")
LOG.debug(r.headers)
LOG.debug(r.content)
with open('reports.csv', "wb") as out_file:
shutil.copyfileobj(r.raw, out_file)
r.content prints the contents of the file transfered correctly.But reports.csv is empty. Also when going to the address from my browser file gets downloaded normally, with contents. What do you think is the the problem?
It worked but only I after I changed the code of while to this according to requests documentation for streaming files.
with open('reports.csv', "wb") as out_file:
for chunk in r.iter_content():
out_file.write(chunk)
Changing the file streamed by the server to a new one doesn't work. I can download the new file from web browser but not from requests and python.
Related
I have a project which I should download some giant files using HTTP and upload them to am FTP server. The simplest way is to first download the file and then upload it to FTP; Thinking it as two independent stages.
But can we use streams to upload the file as it is being downloaded? This way seems more efficient. Any example specially in python are welcome.
Use requests module to obtain a file-like object representing the HTTP download and use it with ftplib FTP.storbinary:
from ftplib import FTP
import requests
url = "https://www.example.com/file.zip"
r = requests.get(url, stream=True)
ftp = FTP(host, user, passwd)
ftp.storbinary("STOR /ftp/path/file.zip", r.raw)
I am building an application where data is sent to a server, the server creates an xlsx (excel) file with that data and returns that file to the client where at the end I want it to be displayed
Im using flask and the creation of the file itself with the data from the client works and the file is saved locally in the same folder. I tried several things but I cant seem to check wether the file was sent back correctly because I dont exactly know how to work with it on the client side. Currently I try sending the file back as following:
return send_file("my_file.xlsx", as_attachment=True)
and I also tried
return send_file("absolute/path/to/my_file", as_attachment=True)
On the client side I also tried all kind of things and Im currently at
print(r.content)
which prints tons of characters, backslashes etc..
and where r is
r = requests.get('http://127.0.0.1:5000/', params = {...})
So two problems:
I dont know if the file is correctly sent from the server, how can I check?
Probably answers the first one: How can I display or save the file on the client side?
The file is created with xlsxwriter and I dont get any error messages. Return status also is 200 so I guess my problem is opening the file on the client side. But if anybody has advice I would be really happy to hear!
Edit: File was sent correctly, the answer was:
r = requests.get('http://127.0.0.1:5000/', params = {...})
def save_xl(r):
with open('file.xlsx', 'wb') as f:
f.write(r.content)
save_xl(r)
And the file was create successfully
you can try saving the content of the request as a xlsx file.
r = requests.get('http://127.0.0.1:5000/', params = {...})
def save_xl(r):
with open('file.xlsx', 'wb') as f:
f.write(r.content)
save_xl(r)
What I am trying to accomplish is to download a specific portion of a video file using python. Sort of what a browser will do when playing a video. If the file is 1000 Bytes, I want to download from byte 200 to 700. I know that I can download the file in parts using the method below:
file_ = open(filename, 'wb')
res = requests.get(url, stream=True)
for chunk in res.iter_content(amount):
file_.write(chunk)
file_.close()
How can I modify this code to accomplish that?
The server has to support this:
If Accept-Ranges is present in HTTP responses (and its value isn't
none), the server supports range requests. You can check this by
issuing a HEAD request.
If the server supports it you can request the part as
curl http://i.imgur.com/z4d4kWk.jpg -i -H "Range: bytes=0-1023"
Source: https://developer.mozilla.org/en-US/docs/Web/HTTP/Range_requests
Also take a look at this.
Is there a way to stream a file directly to the filesystem? Even if the connection is lost, I want to see all contents in that specific file in the fs. Just like wget or curl.
However, using request leads to the issue of first downloading the content of a response and then writing it to a filesystem.
with open(file_name, "wb") as file:
response = get(url) # may take some time
file.write(response.content)
Problem: while the file is "downloading" it is stored elsewhere (I guess in memory or a temporarily splace in the filesystem). That means I have a 0 byte file as long as the request is not (successfully) finished.
Can I solve this problem without using a third party lib?
Streaming directly to file can be achieved with requests and stream=true, or see more useful examples
with open(file_name, 'wb') as f:
with requests.get(url, stream=True) as r:
shutil.copyfileobj(r.raw, f)
I am working on Django, python and app engine, Can anyone please tell me hoe to send a pdf file to a url using urllib2,(file is InMemoryUploadedFile). I know there is a question in SOF for sending data using urllib2 with the data being in JSON format..But here I want to send a InMemoryUploadedFile which is a pdf uploaded file from html page. Thanks in advance...
You might want to look at Python: HTTP Post a large file with streaming.
You will need to use mmap for streaming the file in memory, then set it to the request and set the headers to appropriate mime type i.e application/pdf before opening the url.
import urllib2
import mmap
# Open the file as a memory mapped string. Looks like a string, but
# actually accesses the file behind the scenes.
f = open('somelargefile.pdf','rb')
mmapped_file_as_string = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ)
# Do the request
request = urllib2.Request(url, mmapped_file_as_string)
request.add_header("Content-Type", "application/pdf")
response = urllib2.urlopen(request)
#close everything
mmapped_file_as_string.close()
f.close()
Since Google app engine doesn't have mmap, you might want to write the file in request.FILES to the disk temporarily
#f is the file from request.FILES
def handle_uploaded_file(f):
with open('some/file/name.txt', 'wb+') as destination:
for chunk in f.chunks():
destination.write(chunk)
And then read the file from there directly using standard file operations.
Another option is to use StringIO to write your file to in-memory as a string and then pass it to urlib2.request. This can be inefficient in a multi-user environment compared to using a stream.