I am sending file via requests.post method in python 3.7. The code is something as simple as the following,
with open('filename','rb') as data:
r = requests.post(url, data)
The request is sent to a handler created on AWS Lambda, where the file should then be stored in other services. The body of the event seems to be an encoded string of the file object and I can't find a way to decode it.
Thanks guys!
What you're trying to do is not a great ideia. Lambda has Invocation payload limit of 6MB, so you can't send large files like this.
The best way is to use boto3 appropriate function to upload files directly to S3
If you really want to use requests.post, open the file as a string and send It via post, something like that:
with open('file.txt', 'r') as file:
STRING_FILE = file.read().replace('\n', '')
r = requests.post(<URL>, data = {'key':STRING_FILE})
Related
;TLDR
I want to send a file with requests.send() using multipart/form-data request without storing the file on a hard drive. Basically, I'm looking for an alternative for open() function for bytes object
Hello, I'm currently trying to send multipart/form-data request and pass in-memory files in it, but I can't figure out how to do that.
My app receives images from one source and sends them to another. Currently it sends get request directly to file, (e.g. requests.get('https://service.com/test.jpeg')), reads image's bytes and writes them into new file on the hard drive. The sending code that works looks like this:
def send_file(path_to_image: str)
url = get_upload_link()
data = {'photo': open(path_to_image, 'rb')}
r = requests.post(url, files=data)
send_file("test.jpeg")
The main issue I have with this approach is that I have to keep files on my hard drive. Sure, I can use my drive as some sort of a "temporary buffer" and just delete them after I no longer need these files, but I believe there's much more simple way to do that.
I want my function to receive bytes object and then send it. I actually tried doing that, but the backend doesn't accept them. Here's what I tried to do
Attempt 1
def send_file(image: bytes)
url = get_upload_link()
data = {'photo': open(image, 'rb')}
r = requests.post(url, files=data)
I get "ValueError: embedded null byte"
Attempt 2
def upload_photo(image: bytes):
url = get_upload_link()
file = BytesIO(image)
data = {'photo': file}
r = requests.post(url, files=data)
Backend server doesn't process my files correctly. It's like passing files=None, same response
I also tried:
sending the returning value of the methods: file.getbuffer() and file.read()
file.write(image) and then sending file
StringsIO object
etc.
Final notes
I noticed, that open() returns _io.BufferedReader object. I also looked for a way to construct its instance, but couldn't fund a way. Can someone help me, please?
UPD:
If anyone is interested, the receiving api is this
From the official documentation:
POST a Multipart-Encoded File
...
If you want, you can send strings to
be received as files:
url = 'https://httpbin.org/post'
files = {'file': ('report.csv', 'some,data,to,send\nanother,row,to,send\n')}
r = requests.post(url, files=files)
There's a website that has a button which downloads an Excel file. After I click, it takes around 20 seconds for the server API to generate the file and send it back to my browser for download.
If I monitor the communication after I click the button, I can see how the browser sends a POST request to a server with a series of headers and form values.
Is there a way that I can simulate a similar POST request programmatically using Python, and retrieve the Excel file after the server sends it over?
Thank you in advance
The requests module is used for sending all kinds of request types.
requests.post sends the post requests synchronously.
The payload data can be set using data=
The response can be accessed using .content.
Be sure to check the .status_code and only save on a successful response code
Also note the use of "wb" inside open, because we want to save the file as a binary instead of text.
Example:
import requests
payload = {"dao":"SampleDAO",
"condigId": 1,
...}
r = requests.post("http://url.com/api", data=payload)
if r.status_code == 200:
with open("file.save","wb") as f:
f.write(r.content)
Requests Documentation
I guess You could similarly do this:
file_info = request.get(url)
with open('file_name.extension', 'wb') as file:
file.write(file_info.content)
I honestly do not know how to explain this tho since I have little understanding how it works
I am building an application where data is sent to a server, the server creates an xlsx (excel) file with that data and returns that file to the client where at the end I want it to be displayed
Im using flask and the creation of the file itself with the data from the client works and the file is saved locally in the same folder. I tried several things but I cant seem to check wether the file was sent back correctly because I dont exactly know how to work with it on the client side. Currently I try sending the file back as following:
return send_file("my_file.xlsx", as_attachment=True)
and I also tried
return send_file("absolute/path/to/my_file", as_attachment=True)
On the client side I also tried all kind of things and Im currently at
print(r.content)
which prints tons of characters, backslashes etc..
and where r is
r = requests.get('http://127.0.0.1:5000/', params = {...})
So two problems:
I dont know if the file is correctly sent from the server, how can I check?
Probably answers the first one: How can I display or save the file on the client side?
The file is created with xlsxwriter and I dont get any error messages. Return status also is 200 so I guess my problem is opening the file on the client side. But if anybody has advice I would be really happy to hear!
Edit: File was sent correctly, the answer was:
r = requests.get('http://127.0.0.1:5000/', params = {...})
def save_xl(r):
with open('file.xlsx', 'wb') as f:
f.write(r.content)
save_xl(r)
And the file was create successfully
you can try saving the content of the request as a xlsx file.
r = requests.get('http://127.0.0.1:5000/', params = {...})
def save_xl(r):
with open('file.xlsx', 'wb') as f:
f.write(r.content)
save_xl(r)
I am new to Python and it is first post. Want to upload a TXT file to server, (as of now, it is local host).Every time, I run the script, the local file uploaded and updated on server. I am using Requests module
import requests, time
url ='http://192.168.49.205/test/database/data.txt' # where i want to write
files = {'file':('data.txt','C:\Python27\data.txt','rb')}
#r = requests.post(url,files=files) # this works too
r= requests.post('http://192.168.49.205/test/database/data.txt',
data={'upload_type': 'standard', 'upload_to': '0'},files=files)
print r.status_code
print r.text
data.txt is not updated. Just seeing old data (I put some values when file created).I am not using any Forms in PHP or HTML. Is it possible to upload with method?
Believe,I got some clarity after posting the question. Now, there is PHP file on server side listening to client. Here is the "post.php". So, this will replace the text file on Client side. PHP file get the Name, Task, Value from client and post to "a.txt" on server (local)
<?php
if(isset($_GET["Name"])){
$name=$_GET["Name"];
}
if(isset($_GET["Task"])){
$task=$_GET["Task"];
}
if(isset($_GET["Value"])){
$value=$_GET["Value"];
}
$f=fopen("a.txt","w") or exit("Unable to open file!");
fwrite($f,$name);
fwrite($f," ");
fwrite($f,$task);
fwrite($f," ");
fwrite($f,$value);
fclose($f);
?>`
So now Requests look like this
import requests, time
url = 'http://192.168.49.205/test/test.php'
post_data = {'Name':'job','Task':'008','Value':'8'}
r= requests.post('http://192.168.49.205/test/post.php', data= post_data)
print r.status_code
print r.text
Stil the values is not reaching "a.txt". What i am missing? Please advise!
You're not reading the files content in your code. The requests documentation states:
files – (optional) Dictionary of ‘name’: file-like-objects (or
{‘name’: (‘filename’, fileobj)}) for multipart encoding upload.
Your code suggests you want to use the second option. However:
files = {'file':('data.txt','C:\Python27\data.txt','rb')}
You're not creating the structure as the documentation indicates - you're passing a tuple of 3 strings instead of a tuple of (string, file_obj). You probably wanted to do this:
files = {'file':('data.txt',open('C:\Python27\data.txt','rb'))}
I want to download file to my server and automatically send it to online storage(minus or dropbox) via minus or dropbox API, without saving the downloaded file in my server. So, its like streaming or pipe the HTTP connection. Right now im using minus.com API, but its require file object or local file as parameter. I can't figure out how to convert http response to file object.
It is possible to do this? if possible, how?
concept :
FILE_ON_ANOTHER_SERVER ----(http)---> MY_SERVER ----(http)----> ONLINE_STORAGE
thanks
You can get the data from a response via the read() method
response = urllib2.urlopen(request)
data = response.read()
The variable data has the binary data from the response.
Now you can create a StringIO Object which handles the data as a file like object.
import StringIO
datastream = StringIO.StringIO()
datastream.write(data)
datastream.seek(0)
#create dropbox client
client.put_file('/test', datastream)
urllib2.urlopen(url) will return a file-like object. Can you pass that directly to your minus api? See the urllib2 docs at
http://docs.python.org/library/urllib2