I want to download file to my server and automatically send it to online storage(minus or dropbox) via minus or dropbox API, without saving the downloaded file in my server. So, its like streaming or pipe the HTTP connection. Right now im using minus.com API, but its require file object or local file as parameter. I can't figure out how to convert http response to file object.
It is possible to do this? if possible, how?
concept :
FILE_ON_ANOTHER_SERVER ----(http)---> MY_SERVER ----(http)----> ONLINE_STORAGE
thanks
You can get the data from a response via the read() method
response = urllib2.urlopen(request)
data = response.read()
The variable data has the binary data from the response.
Now you can create a StringIO Object which handles the data as a file like object.
import StringIO
datastream = StringIO.StringIO()
datastream.write(data)
datastream.seek(0)
#create dropbox client
client.put_file('/test', datastream)
urllib2.urlopen(url) will return a file-like object. Can you pass that directly to your minus api? See the urllib2 docs at
http://docs.python.org/library/urllib2
Related
;TLDR
I want to send a file with requests.send() using multipart/form-data request without storing the file on a hard drive. Basically, I'm looking for an alternative for open() function for bytes object
Hello, I'm currently trying to send multipart/form-data request and pass in-memory files in it, but I can't figure out how to do that.
My app receives images from one source and sends them to another. Currently it sends get request directly to file, (e.g. requests.get('https://service.com/test.jpeg')), reads image's bytes and writes them into new file on the hard drive. The sending code that works looks like this:
def send_file(path_to_image: str)
url = get_upload_link()
data = {'photo': open(path_to_image, 'rb')}
r = requests.post(url, files=data)
send_file("test.jpeg")
The main issue I have with this approach is that I have to keep files on my hard drive. Sure, I can use my drive as some sort of a "temporary buffer" and just delete them after I no longer need these files, but I believe there's much more simple way to do that.
I want my function to receive bytes object and then send it. I actually tried doing that, but the backend doesn't accept them. Here's what I tried to do
Attempt 1
def send_file(image: bytes)
url = get_upload_link()
data = {'photo': open(image, 'rb')}
r = requests.post(url, files=data)
I get "ValueError: embedded null byte"
Attempt 2
def upload_photo(image: bytes):
url = get_upload_link()
file = BytesIO(image)
data = {'photo': file}
r = requests.post(url, files=data)
Backend server doesn't process my files correctly. It's like passing files=None, same response
I also tried:
sending the returning value of the methods: file.getbuffer() and file.read()
file.write(image) and then sending file
StringsIO object
etc.
Final notes
I noticed, that open() returns _io.BufferedReader object. I also looked for a way to construct its instance, but couldn't fund a way. Can someone help me, please?
UPD:
If anyone is interested, the receiving api is this
From the official documentation:
POST a Multipart-Encoded File
...
If you want, you can send strings to
be received as files:
url = 'https://httpbin.org/post'
files = {'file': ('report.csv', 'some,data,to,send\nanother,row,to,send\n')}
r = requests.post(url, files=files)
I have a Python REST server, that is able to download and write a temporary file using Python TempFile.
That is, at request time I have a a file in the filesystem of the server, but it is not permanent so the client cannot access it statically (I.e. via http://myserver/path-to-my-file ). I need a way to access an endpoint, and then get a file returned based on the request. (I.e. http://myserver/myendpoint/myrequestparameters)
How does that work over HTTP? Is it possible?
(For context, right now I am serving the file encoded as string using base64 encoding and utf-8 decoding, but my frontend application needs a direct download link)
I believe there's a dedicated response type for such stuff in django. Assuming send_file is your endpoint:
from django.http import FileResponse
def send_file(response):
img = open('my_image_file.jpg', 'rb')
response = FileResponse(img)
return response
I am sending file via requests.post method in python 3.7. The code is something as simple as the following,
with open('filename','rb') as data:
r = requests.post(url, data)
The request is sent to a handler created on AWS Lambda, where the file should then be stored in other services. The body of the event seems to be an encoded string of the file object and I can't find a way to decode it.
Thanks guys!
What you're trying to do is not a great ideia. Lambda has Invocation payload limit of 6MB, so you can't send large files like this.
The best way is to use boto3 appropriate function to upload files directly to S3
If you really want to use requests.post, open the file as a string and send It via post, something like that:
with open('file.txt', 'r') as file:
STRING_FILE = file.read().replace('\n', '')
r = requests.post(<URL>, data = {'key':STRING_FILE})
This is what I'm currently using to send images to the API:
import requests
response = requests.post("http://someAPI",
auth=(username, password),
files={'imgFile': open(filepath, 'rb')}
)
It works for local files, but I would like to be able to send images from a URL as well. I know how to do this by saving the file, then opening it as a local file, sending it to the API and then removing it.
Is there any way of doing this without having to save and remove the file?
You can use StringIO. StringIO is file compatible object which has open,seek,read,write. So you can load data to it and serve from that, without writing file to disk. I normally use this approch for creating CAPTCH.
import StringIO
import requests
# Imagine you read the image file in image as base64 encoded...
buff = StringIO.StringIO()
buff.write(image.decode('base64'))
buff.seek(0)
response = requests.post("http://someAPI",
auth=(username, password),
files={'imgFile': buff}
)
I am working on Django, python and app engine, Can anyone please tell me hoe to send a pdf file to a url using urllib2,(file is InMemoryUploadedFile). I know there is a question in SOF for sending data using urllib2 with the data being in JSON format..But here I want to send a InMemoryUploadedFile which is a pdf uploaded file from html page. Thanks in advance...
You might want to look at Python: HTTP Post a large file with streaming.
You will need to use mmap for streaming the file in memory, then set it to the request and set the headers to appropriate mime type i.e application/pdf before opening the url.
import urllib2
import mmap
# Open the file as a memory mapped string. Looks like a string, but
# actually accesses the file behind the scenes.
f = open('somelargefile.pdf','rb')
mmapped_file_as_string = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ)
# Do the request
request = urllib2.Request(url, mmapped_file_as_string)
request.add_header("Content-Type", "application/pdf")
response = urllib2.urlopen(request)
#close everything
mmapped_file_as_string.close()
f.close()
Since Google app engine doesn't have mmap, you might want to write the file in request.FILES to the disk temporarily
#f is the file from request.FILES
def handle_uploaded_file(f):
with open('some/file/name.txt', 'wb+') as destination:
for chunk in f.chunks():
destination.write(chunk)
And then read the file from there directly using standard file operations.
Another option is to use StringIO to write your file to in-memory as a string and then pass it to urlib2.request. This can be inefficient in a multi-user environment compared to using a stream.